test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots

@ncoghlan

BPO	34207
Nosy	@ncoghlan, @vstinner, @ambv, @koobs, @matrixise, @pablogsal, @tirkarthi
PRs	bpo-34207: Fix pymain_init_cmdline_argv() #8867 bpo-34207: Fix pymain_read_conf() for UTF-8 Mode #8868 [3.7] bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) #8870

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-28.21:10:39.487>
created_at = <Date 2018-07-24.14:08:35.430>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots'
updated_at = <Date 2018-08-28.21:10:39.486>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2018-08-28.21:10:39.486>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-28.21:10:39.487>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2018-07-24.14:08:35.430>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 34207
keywords = ['patch', '3.7regression']
message_count = 12.0
messages = ['322303', '323036', '323037', '323832', '323833', '323834', '323880', '323932', '323935', '323937', '323939', '323940']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'vstinner', 'lukasz.langa', 'koobs', 'matrixise', 'pablogsal', 'xtreak']
pr_nums = ['8867', '8868', '8870']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34207'
versions = ['Python 3.7', 'Python 3.8']

test_cmd_line test_utf8_mode test_warnings fail in AMD64 FreeBSD CURRENT buildbots:

https://buildbot.python.org/all/#/builders/79/builds/186
https://buildbot.python.org/all/#/builders/60/builds/182

Sample error:

test_xoptions (test.test_cmd_line.CmdLineTest) ... ok
test_ignore_PYTHONHASHSEED (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_ignore_PYTHONPATH (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test test_cmd_line failed
test_sys_flags_not_set (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
======================================================================
FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/test_cmd_line.py", line 149, in test_non_ascii
    assert_python_ok('-c', command)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/support/script_helper.py", line 157, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/support/script_helper.py", line 143, in _assert_python
    res.fail(cmd_line)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/Lib/test/support/script_helper.py", line 84, in fail
    err))
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-current.nondebug/build/python', '-X', 'faulthandler', '-I', '-c', "assert(ord('��') == 230)"]
stdout:
---

stderr:
---
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
---
----------------------------------------------------------------------
Ran 43 tests in 17.680s
FAILED (failures=1, skipped=2)
3 tests failed again:
test_cmd_line test_utf8_mode test_warnings

and

est_showwarning_not_callable (test.test_warnings._WarningsTests) ... ok
test_showwarnmsg_missing (test.test_warnings._WarningsTests) ... ok
test test_warnings failed
test_stderr_none (test.test_warnings._WarningsTests) ... ok
======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_warnings/__init__.py", line 1151, in test_nonascii
    "['ignore:Deprecaci��nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_warnings/__init__.py", line 1151, in test_nonascii
    "['ignore:Deprecaci��nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

Ran 111 tests in 4.635s
FAILED (failures=2)
3 tests failed again:
test_cmd_line test_utf8_mode test_warnings
== Tests result: FAILURE then FAILURE ==

I tried the tests on FreeBSD 11.x and FreeBSD 10.x digitalocean droplets with master branch and the tests pass. Adding information that might be helpful.

# FreeBSD 11.2-RELEASE

root@freebsd-s-1vcpu-1gb-blr1-01:~/cpython # uname -a
FreeBSD freebsd-s-1vcpu-1gb-blr1-01 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:32:14 UTC 2018 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

root@freebsd-s-1vcpu-1gb-blr1-01:~/cpython # ./python
Python 3.8.0a0 (heads/master:b6efc2c, Aug 3 2018, 06:26:53)
[Clang 6.0.0 (tags/RELEASE_600/final 326565)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.

>>
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONIOENCODING utf-8
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONWARNINGS ignore:DeprecaciÛnWarning
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONDEVMODE ""
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # ./python -c "import sys; sys.stdout.write(str(sys.warnoptions))"
['ignore:DeprecaciÛnWarning']
root@freebsd-s-1vcpu-1gb-blr1-01:~/cpython # ./python -m unittest -v test.test_warnings test.test_cmd_line test.test_utf8_mode
test_issue_8766 (test.test_warnings.BootstrapTest) ... ok
test_catch_warnings_defaults (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_recording (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_reentry_guard (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_restore (test.test_warnings.CCatchWarningTests) ... ok
test_check_warnings (test.test_warnings.CCatchWarningTests) ... ok
test_record_override_showwarning_before (test.test_warnings.CCatchWarningTests) ... ok
test_record_override_showwarning_inside (test.test_warnings.CCatchWarningTests) ... ok
test_comma_separated_warnings (test.test_warnings.CEnvironmentVariableTests) ... ok
test_conflicting_envvar_and_command_line (test.test_warnings.CEnvironmentVariableTests) ... ok
test_default_filter_configuration (test.test_warnings.CEnvironmentVariableTests) ... ok
test_envvar_and_command_line (test.test_warnings.CEnvironmentVariableTests) ... ok
test_nonascii (test.test_warnings.CEnvironmentVariableTests) ... ok
test_single_warning (test.test_warnings.CEnvironmentVariableTests) ... ok
test_always (test.test_warnings.CFilterTests) ... ok
test_always_after_default (test.test_warnings.CFilterTests) ... ok
test_append_duplicate (test.test_warnings.CFilterTests) ... ok
test_default (test.test_warnings.CFilterTests) ... ok
test_error (test.test_warnings.CFilterTests) ... ok
test_error_after_default (test.test_warnings.CFilterTests) ... ok
test_filterwarnings (test.test_warnings.CFilterTests) ... ok
test_filterwarnings_duplicate_filters (test.test_warnings.CFilterTests) ... ok
test_ignore (test.test_warnings.CFilterTests) ... ok
test_ignore_after_default (test.test_warnings.CFilterTests) ... ok
test_inheritance (test.test_warnings.CFilterTests) ... ok
test_message_matching (test.test_warnings.CFilterTests) ... ok
test_module (test.test_warnings.CFilterTests) ... ok
test_module_globals (test.test_warnings.CFilterTests) ... ok
test_mutate_filter_list (test.test_warnings.CFilterTests) ... ok
test_once (test.test_warnings.CFilterTests) ... ok
test_ordering (test.test_warnings.CFilterTests) ... ok
test_simplefilter_duplicate_filters (test.test_warnings.CFilterTests) ... ok
test_module_all_attribute (test.test_warnings.CPublicAPITests) ... ok
test_improper_input (test.test_warnings.CWCmdLineTests) ... ok
test_accelerated (test.test_warnings.CWarnTests) ... ok
test_bad_str (test.test_warnings.CWarnTests) ... ok
test_exec_filename (test.test_warnings.CWarnTests) ... ok
test_filename (test.test_warnings.CWarnTests) ... ok
test_message (test.test_warnings.CWarnTests) ... ok
test_stacklevel (test.test_warnings.CWarnTests) ... ok
test_stacklevel_import (test.test_warnings.CWarnTests) ... ok
test_warn_explicit_non_ascii_filename (test.test_warnings.CWarnTests) ... ok
test_warn_explicit_type_errors (test.test_warnings.CWarnTests) ... ok
test_warn_nonstandard_types (test.test_warnings.CWarnTests) ... ok
test_warning_classes (test.test_warnings.CWarnTests) ... ok
test_formatwarning (test.test_warnings.CWarningsDisplayTests) ... ok
test_showwarning (test.test_warnings.CWarningsDisplayTests) ... ok
test_finalization (test.test_warnings.FinalizationTest) ... ok
test_late_resource_warning (test.test_warnings.FinalizationTest) ... ok
test_catch_warnings_defaults (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_recording (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_reentry_guard (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_restore (test.test_warnings.PyCatchWarningTests) ... ok
test_check_warnings (test.test_warnings.PyCatchWarningTests) ... ok
test_record_override_showwarning_before (test.test_warnings.PyCatchWarningTests) ... ok
test_record_override_showwarning_inside (test.test_warnings.PyCatchWarningTests) ... ok
test_comma_separated_warnings (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_conflicting_envvar_and_command_line (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_default_filter_configuration (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_envvar_and_command_line (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_nonascii (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_single_warning (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_always (test.test_warnings.PyFilterTests) ... ok
test_always_after_default (test.test_warnings.PyFilterTests) ... ok
test_append_duplicate (test.test_warnings.PyFilterTests) ... ok
test_default (test.test_warnings.PyFilterTests) ... ok
test_error (test.test_warnings.PyFilterTests) ... ok
test_error_after_default (test.test_warnings.PyFilterTests) ... ok
test_filterwarnings (test.test_warnings.PyFilterTests) ... ok
test_filterwarnings_duplicate_filters (test.test_warnings.PyFilterTests) ... ok
test_ignore (test.test_warnings.PyFilterTests) ... ok
test_ignore_after_default (test.test_warnings.PyFilterTests) ... ok
test_inheritance (test.test_warnings.PyFilterTests) ... ok
test_message_matching (test.test_warnings.PyFilterTests) ... ok
test_module (test.test_warnings.PyFilterTests) ... ok
test_module_globals (test.test_warnings.PyFilterTests) ... ok
test_mutate_filter_list (test.test_warnings.PyFilterTests) ... ok
test_once (test.test_warnings.PyFilterTests) ... ok
test_ordering (test.test_warnings.PyFilterTests) ... ok
test_simplefilter_duplicate_filters (test.test_warnings.PyFilterTests) ... ok
test_module_all_attribute (test.test_warnings.PyPublicAPITests) ... ok
test_improper_input (test.test_warnings.PyWCmdLineTests) ... ok
test_improper_option (test.test_warnings.PyWCmdLineTests) ... ok
test_warnings_bootstrap (test.test_warnings.PyWCmdLineTests) ... ok
test_bad_str (test.test_warnings.PyWarnTests) ... ok
test_exec_filename (test.test_warnings.PyWarnTests) ... ok
test_filename (test.test_warnings.PyWarnTests) ... ok
test_message (test.test_warnings.PyWarnTests) ... ok
test_pure_python (test.test_warnings.PyWarnTests) ... ok
test_stacklevel (test.test_warnings.PyWarnTests) ... ok
test_stacklevel_import (test.test_warnings.PyWarnTests) ... ok
test_warn_explicit_non_ascii_filename (test.test_warnings.PyWarnTests) ... ok
test_warn_explicit_type_errors (test.test_warnings.PyWarnTests) ... ok
test_warn_nonstandard_types (test.test_warnings.PyWarnTests) ... ok
test_warning_classes (test.test_warnings.PyWarnTests) ... ok
test_formatwarning (test.test_warnings.PyWarningsDisplayTests) ... ok
test_showwarning (test.test_warnings.PyWarningsDisplayTests) ... ok
test_tracemalloc (test.test_warnings.PyWarningsDisplayTests) ... ok
test_default_action (test.test_warnings._WarningsTests) ... ok
test_filename_none (test.test_warnings._WarningsTests) ... ok
test_filter (test.test_warnings._WarningsTests) ... ok
test_issue31285 (test.test_warnings._WarningsTests) ... ok
test_issue31411 (test.test_warnings._WarningsTests) ... ok
test_issue31416 (test.test_warnings._WarningsTests) ... ok
test_issue31566 (test.test_warnings._WarningsTests) ... ok
test_onceregistry (test.test_warnings._WarningsTests) ... ok
test_show_warning_output (test.test_warnings._WarningsTests) ... ok
test_showwarning_missing (test.test_warnings._WarningsTests) ... ok
test_showwarning_not_callable (test.test_warnings._WarningsTests) ... ok
test_showwarnmsg_missing (test.test_warnings._WarningsTests) ... ok
test_stderr_none (test.test_warnings._WarningsTests) ... ok
test_argv0_normalization (test.test_cmd_line.CmdLineTest) ... skipped 'bpo-32457 only applies on Windows'
test_builtin_input (test.test_cmd_line.CmdLineTest) ... ok
test_closed_stdout (test.test_cmd_line.CmdLineTest) ... ok
test_del___main__ (test.test_cmd_line.CmdLineTest) ... ok
test_directories (test.test_cmd_line.CmdLineTest) ... ok
test_displayhook_unencodable (test.test_cmd_line.CmdLineTest) ... ok
test_empty_PYTHONPATH_issue16309 (test.test_cmd_line.CmdLineTest) ... ok
test_hash_randomization (test.test_cmd_line.CmdLineTest) ... ok
test_isolatedmode (test.test_cmd_line.CmdLineTest) ... ok
test_large_PYTHONPATH (test.test_cmd_line.CmdLineTest) ... ok
test_no_std_streams (test.test_cmd_line.CmdLineTest) ... ok
test_no_stderr (test.test_cmd_line.CmdLineTest) ... ok
test_no_stdin (test.test_cmd_line.CmdLineTest) ... ok
test_no_stdout (test.test_cmd_line.CmdLineTest) ... ok
test_non_ascii (test.test_cmd_line.CmdLineTest) ... ok
test_optimize (test.test_cmd_line.CmdLineTest) ... ok
test_osx_android_utf8 (test.test_cmd_line.CmdLineTest) ... skipped 'test specific to Mac OS X and Android'
test_output_newline (test.test_cmd_line.CmdLineTest) ... ok
test_pythondevmode_env (test.test_cmd_line.CmdLineTest) ... ok
test_pythonmalloc (test.test_cmd_line.CmdLineTest) ... ok
test_run_code (test.test_cmd_line.CmdLineTest) ... ok
test_run_module (test.test_cmd_line.CmdLineTest) ... ok
test_run_module_bug1764407 (test.test_cmd_line.CmdLineTest) ... ok
test_set_pycache_prefix (test.test_cmd_line.CmdLineTest) ... ok
test_showrefcount (test.test_cmd_line.CmdLineTest) ... ok
test_site_flag (test.test_cmd_line.CmdLineTest) ... ok
test_stdin_readline (test.test_cmd_line.CmdLineTest) ... ok
test_stdout_flush_at_shutdown (test.test_cmd_line.CmdLineTest) ... ok
test_sys_flags_set (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_input (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_output (test.test_cmd_line.CmdLineTest) ... ok
test_undecodable_code (test.test_cmd_line.CmdLineTest) ... ok
test_unknown_options (test.test_cmd_line.CmdLineTest) ... ok
test_unmached_quote (test.test_cmd_line.CmdLineTest) ... ok
test_usage (test.test_cmd_line.CmdLineTest) ... ok
test_verbose (test.test_cmd_line.CmdLineTest) ... ok
test_version (test.test_cmd_line.CmdLineTest) ... ok
test_warnings_filter_precedence (test.test_cmd_line.CmdLineTest) ... ok
test_xdev (test.test_cmd_line.CmdLineTest) ... ok
test_xoptions (test.test_cmd_line.CmdLineTest) ... ok
test_ignore_PYTHONHASHSEED (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_ignore_PYTHONPATH (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_sys_flags_not_set (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... ok
test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok

----------------------------------------------------------------------
Ran 164 tests in 6.439s

# FreeBSD 10.4-RELEASE

root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # uname -a
FreeBSD freebsd-s-1vcpu-1gb-blr1-01 10.4-RELEASE FreeBSD 10.4-RELEASE #0 r324094: Fri Sep 29 01:45:44 UTC 2017 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd6
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # ./python
Python 3.8.0a0 (heads/master:b6efc2c, Aug 3 2018, 06:58:22)
[Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.

>>
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONIOENCODING utf-8
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONWARNINGS ignore:DeprecaciÛnWarning
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # setenv PYTHONDEVMODE ""
root@freebsd-s-1vcpu-1gb-blr1-01:/cpython # ./python -c "import sys; sys.stdout.write(str(sys.warnoptions))"
['ignore:DeprecaciÛnWarning']

# Ran as a separate session on the same machine to ignore environment variables

root@freebsd-s-1vcpu-1gb-blr1-01:~/cpython # ./python -m unittest -v test.test_warnings test.test_cmd_line test.test_utf8_mode
test_issue_8766 (test.test_warnings.BootstrapTest) ... ok
test_catch_warnings_defaults (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_recording (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_reentry_guard (test.test_warnings.CCatchWarningTests) ... ok
test_catch_warnings_restore (test.test_warnings.CCatchWarningTests) ... ok
test_check_warnings (test.test_warnings.CCatchWarningTests) ... ok
test_record_override_showwarning_before (test.test_warnings.CCatchWarningTests) ... ok
test_record_override_showwarning_inside (test.test_warnings.CCatchWarningTests) ... ok
test_comma_separated_warnings (test.test_warnings.CEnvironmentVariableTests) ... ok
test_conflicting_envvar_and_command_line (test.test_warnings.CEnvironmentVariableTests) ... ok
test_default_filter_configuration (test.test_warnings.CEnvironmentVariableTests) ... ok
test_envvar_and_command_line (test.test_warnings.CEnvironmentVariableTests) ... ok
test_nonascii (test.test_warnings.CEnvironmentVariableTests) ... ok
test_single_warning (test.test_warnings.CEnvironmentVariableTests) ... ok
test_always (test.test_warnings.CFilterTests) ... ok
test_always_after_default (test.test_warnings.CFilterTests) ... ok
test_append_duplicate (test.test_warnings.CFilterTests) ... ok
test_default (test.test_warnings.CFilterTests) ... ok
test_error (test.test_warnings.CFilterTests) ... ok
test_error_after_default (test.test_warnings.CFilterTests) ... ok
test_filterwarnings (test.test_warnings.CFilterTests) ... ok
test_filterwarnings_duplicate_filters (test.test_warnings.CFilterTests) ... ok
test_ignore (test.test_warnings.CFilterTests) ... ok
test_ignore_after_default (test.test_warnings.CFilterTests) ... ok
test_inheritance (test.test_warnings.CFilterTests) ... ok
test_message_matching (test.test_warnings.CFilterTests) ... ok
test_module (test.test_warnings.CFilterTests) ... ok
test_module_globals (test.test_warnings.CFilterTests) ... ok
test_mutate_filter_list (test.test_warnings.CFilterTests) ... ok
test_once (test.test_warnings.CFilterTests) ... ok
test_ordering (test.test_warnings.CFilterTests) ... ok
test_simplefilter_duplicate_filters (test.test_warnings.CFilterTests) ... ok
test_module_all_attribute (test.test_warnings.CPublicAPITests) ... ok
test_improper_input (test.test_warnings.CWCmdLineTests) ... ok
test_accelerated (test.test_warnings.CWarnTests) ... ok
test_bad_str (test.test_warnings.CWarnTests) ... ok
test_exec_filename (test.test_warnings.CWarnTests) ... ok
test_filename (test.test_warnings.CWarnTests) ... ok
test_message (test.test_warnings.CWarnTests) ... ok
test_stacklevel (test.test_warnings.CWarnTests) ... ok
test_stacklevel_import (test.test_warnings.CWarnTests) ... ok
test_warn_explicit_non_ascii_filename (test.test_warnings.CWarnTests) ... ok
test_warn_explicit_type_errors (test.test_warnings.CWarnTests) ... ok
test_warn_nonstandard_types (test.test_warnings.CWarnTests) ... ok
test_warning_classes (test.test_warnings.CWarnTests) ... ok
test_formatwarning (test.test_warnings.CWarningsDisplayTests) ... ok
test_showwarning (test.test_warnings.CWarningsDisplayTests) ... ok
test_finalization (test.test_warnings.FinalizationTest) ... ok
test_late_resource_warning (test.test_warnings.FinalizationTest) ... ok
test_catch_warnings_defaults (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_recording (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_reentry_guard (test.test_warnings.PyCatchWarningTests) ... ok
test_catch_warnings_restore (test.test_warnings.PyCatchWarningTests) ... ok
test_check_warnings (test.test_warnings.PyCatchWarningTests) ... ok
test_record_override_showwarning_before (test.test_warnings.PyCatchWarningTests) ... ok
test_record_override_showwarning_inside (test.test_warnings.PyCatchWarningTests) ... ok
test_comma_separated_warnings (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_conflicting_envvar_and_command_line (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_default_filter_configuration (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_envvar_and_command_line (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_nonascii (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_single_warning (test.test_warnings.PyEnvironmentVariableTests) ... ok
test_always (test.test_warnings.PyFilterTests) ... ok
test_always_after_default (test.test_warnings.PyFilterTests) ... ok
test_append_duplicate (test.test_warnings.PyFilterTests) ... ok
test_default (test.test_warnings.PyFilterTests) ... ok
test_error (test.test_warnings.PyFilterTests) ... ok
test_error_after_default (test.test_warnings.PyFilterTests) ... ok
test_filterwarnings (test.test_warnings.PyFilterTests) ... ok
test_filterwarnings_duplicate_filters (test.test_warnings.PyFilterTests) ... ok
test_ignore (test.test_warnings.PyFilterTests) ... ok
test_ignore_after_default (test.test_warnings.PyFilterTests) ... ok
test_inheritance (test.test_warnings.PyFilterTests) ... ok
test_message_matching (test.test_warnings.PyFilterTests) ... ok
test_module (test.test_warnings.PyFilterTests) ... ok
test_module_globals (test.test_warnings.PyFilterTests) ... ok
test_mutate_filter_list (test.test_warnings.PyFilterTests) ... ok
test_once (test.test_warnings.PyFilterTests) ... ok
test_ordering (test.test_warnings.PyFilterTests) ... ok
test_simplefilter_duplicate_filters (test.test_warnings.PyFilterTests) ... ok
test_module_all_attribute (test.test_warnings.PyPublicAPITests) ... ok
test_improper_input (test.test_warnings.PyWCmdLineTests) ... ok
test_improper_option (test.test_warnings.PyWCmdLineTests) ... ok
test_warnings_bootstrap (test.test_warnings.PyWCmdLineTests) ... ok
test_bad_str (test.test_warnings.PyWarnTests) ... ok
test_exec_filename (test.test_warnings.PyWarnTests) ... ok
test_filename (test.test_warnings.PyWarnTests) ... ok
test_message (test.test_warnings.PyWarnTests) ... ok
test_pure_python (test.test_warnings.PyWarnTests) ... ok
test_stacklevel (test.test_warnings.PyWarnTests) ... ok
test_stacklevel_import (test.test_warnings.PyWarnTests) ... ok
test_warn_explicit_non_ascii_filename (test.test_warnings.PyWarnTests) ... ok
test_warn_explicit_type_errors (test.test_warnings.PyWarnTests) ... ok
test_warn_nonstandard_types (test.test_warnings.PyWarnTests) ... ok
test_warning_classes (test.test_warnings.PyWarnTests) ... ok
test_formatwarning (test.test_warnings.PyWarningsDisplayTests) ... ok
test_showwarning (test.test_warnings.PyWarningsDisplayTests) ... ok
test_tracemalloc (test.test_warnings.PyWarningsDisplayTests) ... ok
test_default_action (test.test_warnings._WarningsTests) ... ok
test_filename_none (test.test_warnings._WarningsTests) ... ok
test_filter (test.test_warnings._WarningsTests) ... ok
test_issue31285 (test.test_warnings._WarningsTests) ... ok
test_issue31411 (test.test_warnings._WarningsTests) ... ok
test_issue31416 (test.test_warnings._WarningsTests) ... ok
test_issue31566 (test.test_warnings._WarningsTests) ... ok
test_onceregistry (test.test_warnings._WarningsTests) ... ok
test_show_warning_output (test.test_warnings._WarningsTests) ... ok
test_showwarning_missing (test.test_warnings._WarningsTests) ... ok
test_showwarning_not_callable (test.test_warnings._WarningsTests) ... ok
test_showwarnmsg_missing (test.test_warnings._WarningsTests) ... ok
test_stderr_none (test.test_warnings._WarningsTests) ... ok
test_argv0_normalization (test.test_cmd_line.CmdLineTest) ... skipped 'bpo-32457 only applies on Windows'
test_builtin_input (test.test_cmd_line.CmdLineTest) ... ok
test_closed_stdout (test.test_cmd_line.CmdLineTest) ... ok
test_del___main__ (test.test_cmd_line.CmdLineTest) ... ok
test_directories (test.test_cmd_line.CmdLineTest) ... ok
test_displayhook_unencodable (test.test_cmd_line.CmdLineTest) ... ok
test_empty_PYTHONPATH_issue16309 (test.test_cmd_line.CmdLineTest) ... ok
test_hash_randomization (test.test_cmd_line.CmdLineTest) ... ok
test_isolatedmode (test.test_cmd_line.CmdLineTest) ... ok
test_large_PYTHONPATH (test.test_cmd_line.CmdLineTest) ... ok
test_no_std_streams (test.test_cmd_line.CmdLineTest) ... ok
test_no_stderr (test.test_cmd_line.CmdLineTest) ... ok
test_no_stdin (test.test_cmd_line.CmdLineTest) ... ok
test_no_stdout (test.test_cmd_line.CmdLineTest) ... ok
test_non_ascii (test.test_cmd_line.CmdLineTest) ... ok
test_optimize (test.test_cmd_line.CmdLineTest) ... ok
test_osx_android_utf8 (test.test_cmd_line.CmdLineTest) ... skipped 'test specific to Mac OS X and Android'
test_output_newline (test.test_cmd_line.CmdLineTest) ... ok
test_pythondevmode_env (test.test_cmd_line.CmdLineTest) ... ok
test_pythonmalloc (test.test_cmd_line.CmdLineTest) ... ok
test_run_code (test.test_cmd_line.CmdLineTest) ... ok
test_run_module (test.test_cmd_line.CmdLineTest) ... ok
test_run_module_bug1764407 (test.test_cmd_line.CmdLineTest) ... ok
test_set_pycache_prefix (test.test_cmd_line.CmdLineTest) ... ok
test_showrefcount (test.test_cmd_line.CmdLineTest) ... ok
test_site_flag (test.test_cmd_line.CmdLineTest) ... ok
test_stdin_readline (test.test_cmd_line.CmdLineTest) ... ok
test_stdout_flush_at_shutdown (test.test_cmd_line.CmdLineTest) ... ok
test_sys_flags_set (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_input (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_output (test.test_cmd_line.CmdLineTest) ... ok
test_undecodable_code (test.test_cmd_line.CmdLineTest) ... ok
test_unknown_options (test.test_cmd_line.CmdLineTest) ... ok
test_unmached_quote (test.test_cmd_line.CmdLineTest) ... ok
test_usage (test.test_cmd_line.CmdLineTest) ... ok
test_verbose (test.test_cmd_line.CmdLineTest) ... ok
test_version (test.test_cmd_line.CmdLineTest) ... ok
test_warnings_filter_precedence (test.test_cmd_line.CmdLineTest) ... ok
test_xdev (test.test_cmd_line.CmdLineTest) ... ok
test_xoptions (test.test_cmd_line.CmdLineTest) ... ok
test_ignore_PYTHONHASHSEED (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_ignore_PYTHONPATH (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_sys_flags_not_set (test.test_cmd_line.IgnoreEnvironmentTest) ... ok
test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... ok
test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok

----------------------------------------------------------------------
Ran 164 tests in 15.137s

OK (skipped=2)

Thanks

xtreak

in this case, it's an issue on the buildbot cluster, we have to be sure that we don't have this error again.

and for that, we have to find the origin of the crash on buildbot.

This is definitely dependent on the environment. I can reproduce exactly the same failures on a CentOS 7 box at work, too.

I think the root cause is this: the mentioned tests are invalid when no LC_* or LANG environment variables are present. Michael's sleuthing in bpo-34347 might provide a way for us to fix those.

As a workaround in the mean time, if this is an option on your system, set LC_ALL=en_US.UTF8, the three failing tests pass now.

Looking at what's going on here, this is quite possibly a PEP-538 side-effect (I see surrogate escapes).

Nothing has changed recently in the PEP-538 implementation itself, and both those builders were green last month:

The first failing debug build was https://buildbot.python.org/all/#/builders/60/builds/182 which rearranged some parts of the startup sequence: d19d8d5

Similarly, the 3.7 builds started failing in https://buildbot.python.org/all/#/builders/124/builds/489, which made Py_Initialize work more closely to the way it worked in Python 3.6: 0c90d6f

So given the output reported in the current build failures, my guess would be that something has gotten out of sequence in the code that has to handle re-encoding and decoding command line values and environment variables that have initially been decoded incorrectly (i.e. as surrogate escaped ASCII).

I confirm that "LC_ALL= LANG= LC_CTYPE= ./python -m test -v test_cmd_line -m test_non_ascii" fails on FreeBSD on the master branch, whereas it pass before commit d19d8d5. This commit introduced the regression.

Thanks for the bisection Nick!

My change avoids modifying global configurable variables, whereas Py_DecodeLocale() and Py_EncodeLocale() depend on Py_UTF8Mode.

I wrote PR 8867 but it doesn't fix all cases, so I abandonned it.

Then I wrote PR 8868 which sets Py_UTF8Mode in pymain_read_conf() and should fix the issue.

The root issue is that we have C functions which depend on the global configuration variables like Py_UTF8Mode :-(

New changeset 89487f5 by Victor Stinner in branch 'master':
bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868)
89487f5

New changeset 80a0eba by Victor Stinner in branch '3.7':
bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) (GH-8870)
80a0eba

The regression should now be fixed in 3.7 and master branches.

Navigation Menu

Repository navigation

test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots #78388

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots #78388

Description

Activity

pablogsal commented on Jul 24, 2018

tirkarthi commented on Aug 3, 2018

matrixise commented on Aug 3, 2018

ambv commented on Aug 21, 2018

ambv commented on Aug 21, 2018

ambv commented on Aug 21, 2018

ncoghlan commented on Aug 22, 2018

vstinner commented on Aug 23, 2018

vstinner commented on Aug 23, 2018

vstinner commented on Aug 23, 2018

vstinner commented on Aug 23, 2018

vstinner commented on Aug 23, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions