TextIOWrapper: issues with interlaced read-write #56424

Activity

1. vstinner commented on May 30, 2011

   

   vstinner

   on May 30, 2011

   MemberAuthor

   More actions

   The following code fails on an assertion error (Python exception for _pyio, C assertion for io):

   ------------------

   with io.BytesIO(b'abcd') as raw:
       with _pyio.TextIOWrapper(raw, encoding='ascii') as f:
           f.read(1)
           f.write('2')
           f.tell()

   ---

   I found this assertion while testing interlaced read-write on TextIOWrapper:
   ------------------

   with io.BytesIO(b'abcd') as raw:
       with _pyio.TextIOWrapper(raw, encoding='ascii') as f:
           f.write("1")
           # read() must call writer.flush()
           assertEqual(f.read(1), 'b')
           # write() must rewind the raw stream
           f.write('2')
           assertEqual(f.read(), 'd')
           f.flush()
           assertEqual(raw.getvalue(), b'1b2d')
   
   with io.BytesIO(b'abc') as raw:
       with _pyio.TextIOWrapper(raw, encoding='ascii') as f:
           self.assertEqual(f.read(1), b'a')
           # write() must undo reader readahead
           f.write(b"2")
           assertEqual(f.read(1), b'c')
           f.flush()
           assertEqual(raw.getvalue(), b'a2c')

   ---

   These tests fails on "read, write, read" path: write() breaks TextIOWrapper internal state if a read() occured just before. Note that _pyio.TextIOWrapper.write() contains the following comment...

   # XXX What if we were just reading?

   See also the issue bpo-12213 for BufferedRandom and BufferedRWPair.
2. vstinner

   added

   topic-IO

   on May 30, 2011
3. vstinner commented on May 30, 2011

   

   vstinner

   on May 30, 2011

   MemberAuthor

   More actions

   textiowrapper_interlaced_read_write.patch: TextIOWrapper.write() calls self.seek(self.tell()) if it has a decoder or if snapshot is not None.

   I suppose that we can do better, but at least it does fix this issue.

   "read(); write(); write()" only calls self.seek(self.tell()) once, at the first write. So I don't think that it changes anything with performances.

   In which case snapshot can be not different than None, whereas decoder is None?
4. terryjreedy commented on Jun 3, 2011

   

   terryjreedy

   on Jun 3, 2011

   Member

   More actions

   For c stdio files, intermixed reads and writes require a file positioning operation. This is a nuisance and source of program bugs. I do not see any such limitation documented for our io module. So for both reasons, it will be nice to not have the limitation in the code.

   If I understand, the essence of the patch is to do the file positioning automatically internally when needed.
5. vstinner commented on Jun 3, 2011

   

   vstinner

   on Jun 3, 2011

   MemberAuthor

   More actions

   If I understand, the essence of the patch is to do
   the file positioning automatically internally when needed.

   My patch is just a proposition to fix the issue. I wrote "I suppose that we can do better": self.seek(self.tell()) is more a workaround than a real fix. I don't understand why it does fix this bug :-)
6. terryjreedy commented on Jun 4, 2011

   

   terryjreedy

   on Jun 4, 2011

   Member

   More actions

   Perhaps the stdio requirement was based on an underlying OS (*nix?) requirement, which io has to fulfill even if it does not use stdio.

   Stdio was, I presume, optimized for speed. In the relatively rare case of mixed read/write, it *should* put the burden on the programmer. Python is a bit different.
7. BreamoreBoy commented on Jun 24, 2014

   

   BreamoreBoymannequin

   on Jun 24, 2014

   Mannequin

   More actions

   Does anybody want to follow up on this? bpo-12213 was closed as fixed, bpo-12513 is still open.
8. vadmium commented on Jan 20, 2016

   

   vadmium

   on Jan 20, 2016

   Member

   More actions

   Consider codecs that maintain an internal buffer (UTF-7) or other state (ISO-2022). When you call TextIOWrapper.read() and then tell(), the I think the returned number is supposed to hold the _decoder_ state, so you can seek back and read again. But I don’t think the number holds any _encoder_ state, so seek() cannot set up the encoder properly for these more awkward codecs.

   I don’t think it is practical to fix this problem using the incremental codec API. You would need to construct the encoder’s state from the decoder. There are a couple of bugs marked as duplicates of this. What are the real-world use cases?
9. vstinner commented on Jun 28, 2017

   

   vstinner

   on Jun 28, 2017

   MemberAuthor

   More actions

   Given that nobody complains the last 9 years (since Python 3.0 was released), I'm not sure that it's worth it to fix this corner case.

   If you consider that I'm wrong, please reopen the issue ;-)
10. vstinner

    closed this as completedon Jun 28, 2017
11. vadmium commented on Nov 24, 2018

    

    vadmium

    on Nov 24, 2018

    Member

    More actions

    For the record, the more recent bug I mentioned was a complaint from 2015 (one and a half years before Victor’s comment). Even if it is not worth supporting writing after reading, the problem could be documented.
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022