Repository navigation
email package content-transfer-encoding behaviour changed #43702
Description
Activity
ThomasAH commented
on Jul 20, 2006 ThomasAHmannequinMannequinAuthorfrom email.Message import Message from email.Charset import Charset, QP text = "=" msg = Message() charset = Charset("utf-8") charset.header_encoding = QP charset.body_encoding = QP msg.set_charset(charset) msg.set_payload(text) print msg.as_string()
Gives
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable=3D
With the email package from python2.4.3 and 2.4.4c0 the
last '=3D' becomes just '=', so an extra
msg.body_encode(text) is needed.ThomasAH commented
on Jul 20, 2006 ThomasAHmannequinMannequinAuthorLogged In: YES
user_id=839582One program which got hit by this is MoinMoin, see
http://moinmoin.wikiwikiweb.de/MoinMoinBugs/ResetPasswordEmailImproperlyEncodedrdemetrescu commented
on Dec 6, 2007 rdemetrescumannequinMannequinI am not sure if it is related, but anyway...
MIMEText behaviour has changed from python 2.4 to 2.5.
# Python 2.4
>>> from email.MIMEText import MIMEText >>> m = MIMEText(None, 'html', 'iso-8859-1') >>> m.set_payload('abc ' * 50) >>> print m From nobody Thu Dec 6 12:52:40 2007 Content-Type: text/html; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20# Python 2.5
>>> from email.MIMEText import MIMEText >>> m = MIMEText(None, 'html', 'iso-8859-1') >>> m.set_payload('abc ' * 50) >>> print m From nobody Thu Dec 6 14:46:07 2007 Content-Type: text/html; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abcHowever, if we initialize MIMEText with the text, we get the correct output:
# python 2.5
>>> from email.MIMEText import MIMEText >>> m = MIMEText('abc ' * 50, 'html', 'iso-8859-1') >>> print m From nobody Thu Dec 6 13:01:17 2007 Content-Type: text/html; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20If I want to set payload after MIMEText is already created, I need to
use this workaround::#python 2.5 from email.MIMEText import MIMEText m = MIMEText(None, 'html', 'iso-8859-1') m.set_payload(m._charset.body_encode('abc' * 50))
PS: The issue's versions field is filled with "Python 2.4". Shouldn't it
be "Python 2.5" ?paulproteus commented
on Sep 27, 2008 paulproteusmannequinMannequinAnother way to see this issue is that the email module double-encodes
when one attempts to use quoted-printable encoding. This has to be
worked around by e.g. MoinMoin.It's easy to get proper base64-encoded output of email.mime.text:
>>> mt = email.mime.text.MIMEText('Ta mère', 'plain', 'utf-8') >>> 'Content-Transfer-Encoding: base64' in mt.as_string() True >>> mt.as_string().split('\n')[-2] 'VGEgbcOocmU='There we go, all nice and base64'd.
I can *not* figure out how to get quoted-printable-encoding. I found
http://docs.python.org/lib/module-email.encoders.html , so I thought
great - I'll just encode my MIMEText object:>>> email.encoders.encode_quopri(mt) >>> 'Content-Transfer-Encoding: quoted-printable' in mt.as_string() True
Great! Except it's actually double-encoded, and the headers admit to as
much. You see here that, in addition to the quoted-printable header
just discovered, there is also a base64-related header, and the result
is not strictly QP encoding but QP(base64(payload)).>>> 'Content-Transfer-Encoding: base64' in mt.as_string() True >>> mt.as_string().split('\n')[-2] 'VGEgbcOocmU=3D'It should look like:
>>> quopri.encodestring('Ta mère') 'Ta m=C3=A8re'I raised this issue on the Baypiggies list
<http://mail.python.org/pipermail/baypiggies/2008-September/003983.html\>, but
luckily I found this here bug. This is with Python 2.5.2-0ubuntu1 from
Ubuntu 8.04.paulproteus@alchemy:~ $ python --version Python 2.5.2If we can come to a decision as to how this *should* work, I could
contribute a patch and/or tests to fix it. I could even perhaps write a
new section of the Python documentation of the email module explaining this.ThomasAH commented
on May 5, 2010 ThomasAHmannequinMannequinAuthorRoger Demetrescu, I filed the issue with "Python 2.4", because the behavior changed somewhere between 2.4.2 and 2.4.3
The updated link to the MoinMoin bug entry is:
http://moinmo.in/MoinMoinBugs/ResetPasswordEmailImproperlyEncodedThe workaround I use to be compatible with <= 2.4.2 and >= 2.4.3 is:
msg.set_payload('=') if msg.as_string().endswith('='): text = charset.body_encode(text) msg.set_payload(text)
18 remaining items
python-dev commented
on Mar 19, 2013 python-devmannequinMannequinNew changeset ba500b179c3a by R David Murray in branch '3.2':
bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/ba500b179c3aNew changeset fcbc28ef96a3 by R David Murray in branch '3.3':
Merge: bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/fcbc28ef96a3New changeset b9e07f20832e by R David Murray in branch 'default':
Merge: bpo-1525919: Document MIMEText+set_payload encoding behavior.
http://hg.python.org/cpython/rev/b9e07f20832eI've committed the doc change. I'm going to be lazy and leave this issue open to deal with the encodings module fix.
- changed the title
[-]email package quoted printable behaviour changed[/-][+]email package content-transfer-encoding behaviour changed[/+]on Mar 19, 2013 - changed the title
[-]email package quoted printable behaviour changed[/-][+]email package content-transfer-encoding behaviour changed[/+]on Mar 19, 2013 The encoding functions are now doing
orig = msg.get_payload(decode=True)
Does this fix the double-encoding issue?
This change was made in
00ae435ThomasAH commented
on Dec 13, 2021 ThomasAHmannequinMannequinAuthorDefault python3 on Debian buster: $ python3 Python 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import email.mime.text >>> mt = email.mime.text.MIMEText('Ta mère', 'plain', 'utf-8') >>> print(mt.as_string()) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64
VGEgbcOocmU=
>>> email.encoders.encode_quopri(mt) >>> print(mt.as_string()) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Transfer-Encoding: quoted-printableTa=20m=C3=A8re
So the encoded text looks good now, but there are still duplicate headers.
Old output (python2.7) is identical to what Asheesh Laroia (paulproteus)
reported for python2.5:
---
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Transfer-Encoding: quoted-printableVGEgbcOocmU=3D
One piece of this is still broken at main: the duplicate-header half.
email.encoders.encode_base64/encode_quopri/encode_7or8bitall appendContent-Transfer-Encodinginstead of replacing it, so calling any of them on a message that already carries one —MIMETextsets it by default — produces twoContent-Transfer-Encoding:headers in the output. Theget_payload(decode=True)change linked above did fix the payload half (that landed back in bpo-18324), but I still get two CTE headers at 3.16 on aMIMETextpayload, so the duplicate header is what's left. The fix is one line per encoder —del msg['Content-Transfer-Encoding']before the assignment (delon a missing header is a no-op, so the normal path is unchanged). I can send the patch with tests.Reproducer at current main (6df4993)
"""gh-43702: the three email.encoders append a second Content-Transfer-Encoding header when the message already carries one (MIMEText sets one by default).""" import email.mime.text, email.encoders mt = email.mime.text.MIMEText('héllo wörld', 'plain', 'utf-8') email.encoders.encode_quopri(mt) ctes = [h for h in mt.keys() if h == 'Content-Transfer-Encoding'] print('encode_quopri CTE count:', len(ctes)) # -> 2 (base64 + quoted-printable) print('payload ok (not double-enc):', mt.get_payload()) # -> placeholder for enc in ('encode_base64', 'encode_7or8bit'): m = email.mime.text.MIMEText('hi ' * 3, 'plain', 'utf-8') getattr(email.encoders, enc)(m) print(enc, 'CTE count:', len([h for h in m.keys() if h == 'Content-Transfer-Encoding'])) # -> 2 # Fix: `del msg['Content-Transfer-Encoding']` before assigning (no-op when absent). mt = email.mime.text.MIMEText('héllo wörld', 'plain', 'utf-8') orig = mt.get_payload(decode=True) mt.set_payload(email.encoders._qencode(orig)) del mt['Content-Transfer-Encoding'] mt['Content-Transfer-Encoding'] = 'quoted-printable' print('fixed CTE count:', len([h for h in mt.keys() if h == 'Content-Transfer-Encoding'])) # -> 1
encode_quopri CTE count: 2 payload ok (not double-enc): h=C3=A9llo=20w=C3=B6rld encode_base64 CTE count: 2 encode_7or8bit CTE count: 2 fixed CTE count: 1
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs