Repository navigation
packaging uses wrong line endings in RECORD files on Windows #57384
Description
| BPO | 13175 |
|---|---|
| Nosy | @pfmoore, @vsajip, @tarekziade, @merwok |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/merwok'
closed_at = <Date 2014-03-12.09:45:46.363>
created_at = <Date 2011-10-13.22:47:01.617>
labels = ['type-bug']
title = 'packaging uses wrong line endings in RECORD files on Windows'
updated_at = <Date 2014-03-12.09:45:46.361>
user = 'https://github.com/pfmoore'bugs.python.org fields:
activity = <Date 2014-03-12.09:45:46.361>
actor = 'eric.araujo'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2014-03-12.09:45:46.363>
closer = 'eric.araujo'
components = ['Distutils2']
creation = <Date 2011-10-13.22:47:01.617>
creator = 'paul.moore'
dependencies = ['13198']
files = ['23416']
hgrepos = []
issue_num = 13175
keywords = ['patch']
message_count = 17.0
messages = ['145488', '145533', '145553', '145595', '145597', '145619', '145620', '145688', '145700', '145723', '152961', '153011', '153028', '153105', '153118', '161035', '213232']
nosy_count = 5.0
nosy_names = ['paul.moore', 'vinay.sajip', 'tarek', 'eric.araujo', 'alexis']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13175'
versions = ['3rd party', 'Python 3.3']
Activity
pfmoore commentedon Oct 13, 2011
On Windows, packaging seems to create RECORD files with an additional CR at the end of the line. (So the line end is CR CR LF). This does not seem to be consistent, but it is likely to be because a file is being opened in text mode rather than binary.
I am trying to develop a reproducible test case, but am having difficulty at the moment. I have opened this bug in any case as a place holder and in case someone else can reproduce the issue.
merwok commentedon Oct 14, 2011
packaging.util creates a RECORD file with OS-specific line endings and csv.writerow; packaging.command.install_distinfo uses LF everywhere and csv.writerow. I’m not sure this is related to binary vs. text I/O; maybe it’s some other code like the RESOURCES system that’s giving lines with endings to install_distinfo, which then adds CRLF. Do you have a RECORD file to upload for inspection?
pfmoore commentedon Oct 14, 2011
Unfortunately, no. I have been unable to get this in a reproducible form - but I have seen it a few times now. I will keep trying to reproduce.
The worst thing is that packaging fails to recognise the data in RECORD and won't uninstall the package. It would be helpful if packaging were a little more permissive in how it handles this sort of problem in RECORD.
vsajip commentedon Oct 15, 2011
I'm trying to reproduce this too, but have failed so far. Which version(s) of Windows did the problems appear on?
pfmoore commentedon Oct 15, 2011
Windows 7, 32 bit.
pfmoore commentedon Oct 16, 2011
I found the problem - it's in packaging.util.write_record_file. The
file passed to csv.writer should be opened with newline=''.
pfmoore commentedon Oct 16, 2011
Here's a patch. It includes a test, but I don't expect the test will catch the issue except on Windows...
merwok commentedon Oct 17, 2011
pfmoore commentedon Oct 17, 2011
On 17 October 2011 14:15, Éric Araujo <report@bugs.python.org> wrote:
No idea :-( The 2.7 documentation says use the 'b' flag, but that
probably doesn't allow an encoding parameter (it doesn't on 3.x).
The test fails before the fix, passes after. It's a no-op on platforms
where text and binary files are the same, (i.e., non-Windows systems).
So it's harmless.
merwok commentedon Oct 17, 2011
Ah, I see that newline controls newline translation. We use codecs.open in distutils2; I think it uses universal newlines mode. We’ll figure it out, the important part is to have good tests.
Great. I’ll look at your patch and try to port it to distutils2.
pfmoore commentedon Feb 9, 2012
Any progress on this one?
I tried to reproduce using distutils2, but don't know how distutils2 works - there's no pysetup and python -m distutils2.run doesn't work the same as python -m packaging.run.
Can we at least get packaging working in time for 3.3, even if it leaves an issue with distutils2 to sort out?
merwok commentedon Feb 10, 2012
Maybe you’re using the old PyPI release or Tarek’s outdated Bitbucket repository. http://hg.python.org/distutils is current, contains pysetup and supports -m distutils2.run.
I’ll port your patch to d2 this week.
pfmoore commentedon Feb 10, 2012
I just did pip install distutils2 in a virtualenv. If that doesn't
work then yes, I got the wrong one :-(
Do I need to hg clone then do some magic pip incantation to get it
installed? I'm new to pip/viirtualenv. I'll have a look.
merwok commentedon Feb 11, 2012
FTR, “bin/pip install -e .” is the command to “install” a project in develop (or editable) mode with pip in a virtualenv.
pfmoore commentedon Feb 11, 2012
Thanks, that's what I needed. I'm new to pip and know little more than
"pip install" :-)
merwok commentedon May 18, 2012
I should be able to run a VM to test and commit this soon. Keep faith!
merwok commentedon Mar 12, 2014
This issue is now obsolete for distutils2. Other projects that write RECORD files are hopefully more robust in their code and tests.