Skip to content

gh-107450: Check for overflow in the tokenizer and fix overflow test #110832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lysnikolaou merged 6 commits into from
Oct 16, 2023
Merged

gh-107450: Check for overflow in the tokenizer and fix overflow test #110832

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
16d508d
gh-107450: Check for overflow in the tokenizer and fix overflow test
lysnikolaou Oct 13, 2023
c8433ca
Address some feedback
lysnikolaou Oct 16, 2023
e4a059e
Address more feedback
lysnikolaou Oct 16, 2023
5d7c5a6
Formatting
lysnikolaou Oct 16, 2023
66d94be
More feedback
lysnikolaou Oct 16, 2023
f20b95f
Use bytes for source
lysnikolaou Oct 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address more feedback
  • Loading branch information
@lysnikolaou
lysnikolaou committed Oct 16, 2023
commit e4a059e1db318425c24748afa4618f80bec89217
9 changes: 2 additions & 7 deletions Lib/test/test_exceptions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Python test set -- part 5, built-in exceptions

import copy
import ctypes
import os
import sys
import unittest
Expand All @@ -10,7 +9,7 @@
import errno
from textwrap import dedent

from _testcapi import INT_MAX
from _testcapi import INT_MAX, PY_SSIZE_T_MAX
Comment thread
lysnikolaou marked this conversation as resolved.
Outdated
from test.support import (captured_stderr, check_impl_detail,
cpython_only, gc_collect,
no_tracing, script_helper,
Expand Down Expand Up @@ -320,19 +319,15 @@ def baz():
check('(yield i) = 2', 1, 2)
check('def f(*):\n pass', 1, 7)

@unittest.skipIf(ctypes.sizeof(ctypes.c_int) >= ctypes.sizeof(ctypes.c_ssize_t),
"Downcasting to int is safe for col_offset")
@unittest.skipIf(INT_MAX >= PY_SSIZE_T_MAX, "Downcasting to int is safe for col_offset")
@support.requires_resource('cpu')
@support.bigmemtest(INT_MAX, memuse=2, dry_run=False)
def testMemoryErrorBigSource(self, size):
Comment thread
lysnikolaou marked this conversation as resolved.
padding_needed = INT_MAX-len("pass")
if size < padding_needed:
Comment thread
lysnikolaou marked this conversation as resolved.
Outdated
self.skipTest('Not enough memory for overflow to occur')

# Construct buffer to hold just enough characters so that the tokenizer offset overflows.
# This makes sure that we don't overflow in the string creation itself
src = f"if True:\n{' ' * padding_needed}pass"

with self.assertRaisesRegex(OverflowError, "Parser column offset overflow"):
compile(src, '<fragment>', 'exec')

Expand Down