mailbox.mbox fails on non ASCII characters

@warsaw

BPO	42433
Nosy	@warsaw, @terryjreedy, @bitdancer, @flokli
PRs	bpo-28764: mailbox.mbox: handle lines with non-ascii more graceful #23553

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-11-22.10:09:32.634>
labels = ['type-feature', 'expert-email', '3.10']
title = 'mailbox.mbox fails on non ASCII characters'
updated_at = <Date 2020-12-05.20:44:11.920>
user = 'https://github.com/flokli'

bugs.python.org fields:

activity = <Date 2020-12-05.20:44:11.920>
actor = 'terry.reedy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2020-11-22.10:09:32.634>
creator = 'flokli'
dependencies = []
files = []
hgrepos = []
issue_num = 42433
keywords = ['patch']
message_count = 8.0
messages = ['381607', '381627', '381637', '381975', '382052', '382169', '382581', '382584']
nosy_count = 4.0
nosy_names = ['barry', 'terry.reedy', 'r.david.murray', 'flokli']
pr_nums = ['23553']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue42433'
versions = ['Python 3.10']

I'm importing some mbox archives into my maildirs, and use mailbox.mbox to parse archives created by pipermail.

Some of these archives seem to contain non-ascii characters, and python just throws a UnicodeDecodeError and refuses to process the archive.

Reproducer: (successful on 3.7.9, 3.8.5, 3.9.0)

curl https://lists.freedesktop.org/archives/systemd-devel/2016-January.txt.gz | zcat > mbox.txt
python3 -c "import mailbox; mb = mailbox.mbox('mbox.txt');mb.items()"

The problem with that archive is that it is not in proper mbox format. It contains the following line (5689):

From here I was hoping to run something like “dbus-send –system –dest=Test.Me –print-reply /Japan Japan.Reset.Test string:”Hello””

You will note that there is no leading '>' on that line to escape that 'From '. So mbox tries to build a 'From ' line from it, and fails because 'From ' lines should not contain any non-ascii characters. It can be argued that that failure is sub-optimal...it should probably be calling decode('ascii', errors='replace') so that the parse doesn't fail, just like it would not fail if there were no non-ascii in the unescaped 'From ' line.

Yeah, not questioning here this might be badly formatted, but given these files are out there, and the parser is somewhat forgiving in other cases, it should be tolerant there as well.

Based on https://bugs.python.org/issue42433#msg382169 I added back the versions that bug is present.

The PR is up to and appropriately linked (I think?) - let me know if there's anything left to be done from my side.

3.7 only gets security fixes. If and when someone merges something, that person will decide whether to backport.

Navigation Menu

Repository navigation

mailbox.mbox fails on non ASCII characters #86599

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

mailbox.mbox fails on non ASCII characters #86599

Description

Activity

flokli commented on Nov 22, 2020

bitdancer commented on Nov 22, 2020

flokli commented on Nov 22, 2020

9 remaining items

flokli commented on Dec 5, 2020

terryjreedy commented on Dec 5, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions