Skip to content

bpo-43260: io: Prevent large data remains in textio buffer. #24592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
methane merged 8 commits into from
Feb 21, 2021
Merged

bpo-43260: io: Prevent large data remains in textio buffer. #24592

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3ba68b0
bpo-43260: io: Prevent large data remains in textio.
methane Feb 20, 2021
5ffbe8f
Add news entry
methane Feb 20, 2021
ccda9bc
Use chunk_size instead of heuristic 1MiB
methane Feb 20, 2021
a17a9f4
Use Eryk's idea
methane Feb 20, 2021
54ff77e
fix comment
methane Feb 20, 2021
960b928
<=
methane Feb 20, 2021
9572c1e
concatenate
methane Feb 20, 2021
05ab6e0
Add test
methane Feb 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use Eryk's idea
  • Loading branch information
@methane
methane committed Feb 20, 2021
commit a17a9f41345c782b64315e46db741c39e9d94c31
36 changes: 16 additions & 20 deletions Modules/_io/textio.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1644,7 +1644,10 @@ _io_TextIOWrapper_write_impl(textio *self, PyObject *text)

/* XXX What if we were just reading? */
if (self->encodefunc != NULL) {
if (PyUnicode_IS_ASCII(text) && is_asciicompat_encoding(self->encodefunc)) {
if (PyUnicode_IS_ASCII(text) &&
// See bpo-43260
PyUnicode_GET_LENGTH(text) < self->chunk_size &&
is_asciicompat_encoding(self->encodefunc)) {
Comment thread
methane marked this conversation as resolved.
b = text;
Py_INCREF(b);
}
Expand All @@ -1653,8 +1656,9 @@ _io_TextIOWrapper_write_impl(textio *self, PyObject *text)
}
self->encoding_start_of_stream = 0;
}
else
else {
b = PyObject_CallMethodOneArg(self->encoder, _PyIO_str_encode, text);
}

Py_DECREF(text);
if (b == NULL)
Expand All @@ -1675,26 +1679,18 @@ _io_TextIOWrapper_write_impl(textio *self, PyObject *text)
bytes_len = PyBytes_GET_SIZE(b);
}

// bpo-43260: If `b` is large, leaving it in pending_bytes may cause
// MemoryError and we can not recover from it forever.
// So do not leave the large data in pending_bytes buffer.
if (bytes_len > self->chunk_size) {
if (_textiowrapper_writeflush(self)) {
Py_DECREF(b);
return NULL;
}
if (self->pending_bytes == NULL) {
self->pending_bytes_count = 0;
self->pending_bytes = b;
self->pending_bytes_count = bytes_len;
if (_textiowrapper_writeflush(self)) {
Py_CLEAR(self->pending_bytes);
self->pending_bytes_count = 0;
}
else if (self->pending_bytes_count + bytes_len > self->chunk_size) {
// bpo-43260: If `b` is very large, leaving it in pending_bytes may
// cause MemoryError and we can not recover from it forever.
// So do not leave the large data in pending_bytes buffer.
if (_textiowrapper_writeflush(self) < 0) {
Comment thread
methane marked this conversation as resolved.
Py_DECREF(b);
return NULL;
}
return PyLong_FromSsize_t(textlen);
}

if (self->pending_bytes == NULL) {
self->pending_bytes_count = 0;
self->pending_bytes = b;
}
else if (!PyList_CheckExact(self->pending_bytes)) {
Expand All @@ -1716,7 +1712,7 @@ _io_TextIOWrapper_write_impl(textio *self, PyObject *text)
}

self->pending_bytes_count += bytes_len;
if (self->pending_bytes_count > self->chunk_size || needflush ||
if (self->pending_bytes_count >= self->chunk_size || needflush ||
Comment thread
methane marked this conversation as resolved.
text_needflush) {
if (_textiowrapper_writeflush(self) < 0)
return NULL;
Expand Down