CSVReader ignores dialect.lineterminator #81890

Activity

1. BenjaminSchollnick commented on Jul 29, 2019

   

   BenjaminSchollnickmannequin

   on Jul 29, 2019

   MannequinAuthor

   More actions

   I've run into a situation where the CSV input file is very unusual. The Delimiter is "\x06" and the lineterminator is "\x07".

   While I've written code to work around this, it would be significantly nicer if the CSV Reader code actually paid attention to the dialect's lineterminator value.
2. BenjaminSchollnick

   added

   3.7 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-featureA feature request or enhancementA feature request or enhancement

   on Jul 29, 2019
3. tirkarthi commented on Jul 29, 2019

   

   tirkarthi

   on Jul 29, 2019

   Member

   More actions

   Seems related : https://bugs.python.org/issue1072404 . There is a note on docs that it's ignored and may be changed in future.
   https://docs.python.org/3/library/csv.html
4. smontanaro commented on Jul 30, 2019

   

   smontanaro

   on Jul 30, 2019

   Contributor

   More actions

   I imagine this is a corner case which will continue to cause problems. At the time the csv module was originally written, I believe the authors' intent was to read and write CSV files which were compatible with Excel. In Python 3, you have to open input files in text mode (that provides the underlying line splitting behavior). Consequently, you're not going to see proper line splitting with unadorned files.

   Have you only tried this with Python 3? If you have tried Python 2, were you able to get it to work without your workaround?
5. BenjaminSchollnick commented on Jul 30, 2019

   

   BenjaminSchollnickmannequin

   on Jul 30, 2019

   MannequinAuthor

   More actions

   This is tested under python 3...

   filename = "csv_Sample.csv"
   from csv import DictReader
   datafile = open(filename, 'r')
   data = csv.DictReader(datafile, lineterminator = '\x07', delimiter='\x06')
   print(next(data))
       OrderedDict([('Field1', 'A'), ('Field2', 'B'), ('Field3', 'C'), ('Field4', 'D'), ('Field5', 'E'), ('Field6', 'F'), ('Field7', 'G'), ('Field8', 'H'), ('Field9', 'I'), ('Field10\x07', 'J\x07')])
   print(ord(data.reader.dialect.lineterminator))

   So it's untested under python 2, since I've stopped developing under Py2.

   I noticed the note in the CSV reader documentation, *AFTER* I diagnosed the issue with the CSV reader... Which is why I opened the bug / feature enhancement request, since this is an very odd edge case.

   I agree 90+% of all CSVs are going to be \n line terminated, but if we offer it for writing, we should offer it for reading.

   The main emphasis here is this code will not working in the real world, eg.

   filename = "csvFile.csv"
   from csv import DictReader, DictWriter
   import csv
   with open(filename, mode='w') as output_file:
       outcsv = csv.writer(output_file, delimiter=',', lineterminator=";")
       outcsv.writerow(['John Cleese', 'CEO', 'March'])
       outcsv.writerow(['Graham Chapman', 'CFO', 'November'])
       outcsv.writerow(['Terry Jones', 'Animation', 'March'])
       outcsv.writerow(['Eric Idle', 'Laugh Track', 'November'])
       outcsv.writerow(['Michael Palin', 'Snake Wrangler', 'March'])
   
   with open(filename, mode='r') as input_file:
       csv_reader = csv.reader(input_file, delimiter=',', lineterminator=";")
       for row in csv_reader:
           print(row)
6. smontanaro commented on Jul 31, 2019

   

   smontanaro

   on Jul 31, 2019

   Contributor

   More actions

   Looking at your sample file, it seems stranger than you first indicated. Your line terminator actually appears to be '\x07\r\n', not just '\x07'. Opening your file in text mode will leave you with '\x07' as the last character of the last cell in each row. I've attached two files, bell.csv, which has just '\x07' as the line terminator, and lfmapper.py, which provides a class (suboptimally named LFMapper) which takes a file object opened in binary mode and optional line_terminator and encoding args, and performs the necessary slicing of the input bytes, decoding them and returning strings.

   Unless Python grows a way for you to tell the open() function what string to use as the line terminator in text mode, I don't think your example is ever going to work without some sort of shim class.
7. DanielSmejkal

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   topic-2to3

   topic-email

   on Jul 22, 2020
8. DanielSmejkal

   assigned

   docspython

   on Jul 22, 2020
9. iritkatriel

   closed this as completedon Oct 20, 2021
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022