bytes-like objects with socket.sendall(), SSL, and http.client

@tiran

BPO	27340
Nosy	@tiran, @vadmium
PRs	bpo-27340: Use memoryview in SSLSocket.sendall() #3384 [3.6] bpo-27340: Use memoryview in SSLSocket.sendall() (GH-3384) #3434

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-08.00:05:19.262>
created_at = <Date 2016-06-17.04:25:20.448>
labels = ['3.7', 'expert-SSL', 'type-bug', 'library', 'docs']
title = 'bytes-like objects with socket.sendall(), SSL, and http.client'
updated_at = <Date 2017-09-08.00:05:19.261>
user = 'https://github.com/vadmium'

bugs.python.org fields:

activity = <Date 2017-09-08.00:05:19.261>
actor = 'christian.heimes'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-09-08.00:05:19.262>
closer = 'christian.heimes'
components = ['Documentation', 'Library (Lib)', 'SSL']
creation = <Date 2016-06-17.04:25:20.448>
creator = 'martin.panter'
dependencies = []
files = []
hgrepos = []
issue_num = 27340
keywords = ['patch']
message_count = 4.0
messages = ['268707', '301639', '301652', '301653']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'docs@python', 'martin.panter']
pr_nums = ['3384', '3434']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27340'
versions = ['Python 3.6', 'Python 3.7']

According to the documentation, HTTPSConnection.request() should accept arbitrary bytes-like objects, but this is not the case. Currently (since bpo-23756), a “bytes-like object” is defined to be anything that works with Python’s buffer API, as long as it is C-contiguous. These objects can be passed to socket.sendall():

>> byteslike = (ctypes.c_ubyte * 6).from_buffer_copy(b"DATA\r\n")
>> s = socket.create_connection(("localhost", 80))
>> s.sendall(byteslike) # Server receives b"DATA\r\n"

This is not explicitly documented for socket objects. But since bpo-23539 (3.4+), HTTPConnection.request() does document support for bytes-like objects:

>> h = HTTPConnection("localhost", 80)
>> # Send Content-Length: 6 and body b"DATA\r\n"
>> h.request("POST", "/", body=byteslike)

On its own, there is no problem with Python relying on its own undocumented behaviour. But Python’s “ssl” module does not support arbitrary bytes-like objects, and as a result neither does HTTPSConnection:

>>> s = ssl.wrap_socket(socket.create_connection(("localhost", 443)))
>>> s.sendall(byteslike)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'
>>> c = ssl.create_default_context(cafile="/lib/python3.5/test/keycert.pem")
>>> h = HTTPSConnection("localhost", 443, context=c)
>>> h.request("POST", "/", body=byteslike)
Traceback (most recent call last):
  File "/usr/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 892, in send
    "or an iterable, got %r" % type(data))
TypeError: data should be a bytes-like object or an iterable, got <class '__main__.c_ubyte_Array_6'>

This could be fixed in the implementation of SSLSocket.sendall(). But I am not sure if that is an appropriate change for 3.5. Another option would be to adjust the documentation of HTTP(S)Connection in 3.5, either not mentioning bytes-like objects at all, or clarifying that they don’t work with SSL.

New changeset 888bbdc by Christian Heimes in branch 'master':
bpo-27340: Use memoryview in SSLSocket.sendall() (bpo-3384)
888bbdc

New changeset 9423f5d by Christian Heimes in branch '3.6':
[3.6] bpo-27340: Use memoryview in SSLSocket.sendall() (GH-3384) (bpo-3434)
9423f5d

In Python 3.6 and master, SSLSocket.sendall() now accepts any bytes-like object that can be cast to bytes. Since it uses a sliced memoryview instead of sliced bytes, it avoids extra copies of the payload, too.

Navigation Menu

Repository navigation

bytes-like objects with socket.sendall(), SSL, and http.client #71527

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bytes-like objects with socket.sendall(), SSL, and http.client #71527

Description

Activity

vadmium commented on Jun 17, 2016

tiran commented on Sep 7, 2017

tiran commented on Sep 7, 2017

tiran commented on Sep 8, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions