Skip to content

BufferedRandom: issues with interlaced read-write #56422

Closed
Closed
BufferedRandom: issues with interlaced read-write#56422
Labels
topic-IOtype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on May 30, 2011
BPO 12213
Nosy @jcea, @pitrou, @vstinner
Files
  • io_interlaced_read_write.patch
  • bufrandom.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2011-08-20.17:56:47.589>
created_at = <Date 2011-05-30.02:52:39.072>
labels = ['type-bug', 'expert-IO']
title = 'BufferedRandom: issues with interlaced read-write'
updated_at = <Date 2011-09-04.06:41:30.775>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2011-09-04.06:41:30.775>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2011-08-20.17:56:47.589>
closer = 'pitrou'
components = ['IO']
creation = <Date 2011-05-30.02:52:39.072>
creator = 'vstinner'
dependencies = []
files = ['22190', '22950']
hgrepos = []
issue_num = 12213
keywords = ['patch']
message_count = 15.0
messages = ['137238', '137239', '137259', '137261', '137690', '138736', '142479', '142487', '142528', '142529', '142530', '142532', '142545', '142546', '143476']
nosy_count = 6.0
nosy_names = ['jcea', 'pitrou', 'vstinner', 'Arfrever', 'santoso.wijaya', 'python-dev']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12213'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    Activity

    1. vstinner commented on May 30, 2011

      @vstinner
      vstinner
      on May 30, 2011
      MemberAuthor

      The following code displays "Xbc" using io, and "bc" using _pyio (or an unbuffered file, e.g. io.FileIO):
      -------------

      import _pyio, io

with io.BytesIO(b'abc') as raw:
    #with _pyio.BufferedRandom(raw) as f:
    with io.BufferedRandom(raw) as f:
        f.write(b"X")
        print("pos?", f.tell(), raw.tell())
        print(f.read())

      I expect .write() to change the file position, and so "bc" must be the correct result, not "Wbc".

      _pyio.BufferedRandom overrides its write method, whereas io.BufferedRandom doesn't.

      I already noticed the implement difference of BufferedRandom.write(), but I don't remember if I reported it or not!?

    2. vstinner
      added
      topic-IO
      on May 30, 2011

    3. vstinner commented on May 30, 2011

      @vstinner
      vstinner
      on May 30, 2011
      MemberAuthor

      I already noticed the implement difference of BufferedRandom.write(),
      but I don't remember if I reported it or not!?

      Yes, I already reported the difference:
      http://bugs.python.org/issue12062#msg135876

    4. vstinner
      changed the title [-]BufferedRandom: write(); read() gives different result using io and _pyio[/-] [+]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/+] on May 30, 2011

    5. vstinner commented on May 30, 2011

      @vstinner
      vstinner
      on May 30, 2011
      MemberAuthor

      io_interlaced_read_write.patch:

      • add interlaced read/write tests for BufferedRandom and BufferedRWPair
      • _pyio: move "undo readahead" code into BufferedReader.flush()
      • io: BufferedRandom.flush() doesn't undo readahead if the write buffer is empty, so it's possible to call it in read methods without flusing the readahead buffer when it's not needed
      • read(), read1(), readinto(), peek() calls writer.flush()
      • write() calls reader.flush() (undo readahead)

      TODO:

      • tests BufferedRWPair with read-only + write-only files because I'm not sure if my tests using readable and writeable methods
      • _pyio: undo the readahead in BufferedReader.flush() is no perfect, because BufferedReader is supposed to be read-only. I choosed that to factorize the code between BufferedRandom and BufferedRWPair
      • what happens if a write occurs during _pyio.BufferedReader.flush()? "if self._read_buf: <write occurs> with self._read_lock: ...". We may protect the read of self._read_buf with the read lock.

      My patch tries to fix interlaced read-write by always calling flush(), but I am not sure that it doesn't change read-only and write-only cases. There are maybe some unnecessary call to flush().

    6. vstinner commented on May 30, 2011

      @vstinner
      vstinner
      on May 30, 2011
      MemberAuthor

      See also issue bpo-12215: TextIOWrapper has also issues on interlaced read-write.

    7. pitrou commented on Jun 5, 2011

      @pitrou
      pitrou
      on Jun 5, 2011
      Member

      My patch tries to fix interlaced read-write by always calling flush(),

      Why do you need to call flush()? Can't you read from the buffer?

    8. vstinner commented on Jun 20, 2011

      @vstinner
      vstinner
      on Jun 20, 2011
      MemberAuthor

      > My patch tries to fix interlaced read-write by always calling flush(),

      Why do you need to call flush()? Can't you read from the buffer?

      Hum, my patch does not always call flush of the reader and the writer. On read, it flushs the writer. On write, it "flushes" the reader (undo readahead).

      It is maybe possible to do better (do something faster), but there should be some tricky cases with seek().

    9. pitrou
      added
      type-bugAn unexpected behavior, bug, or error
      on Jul 1, 2011

    10. pitrou commented on Aug 19, 2011

      @pitrou
      pitrou
      on Aug 19, 2011
      Member

      I think your expectations for BufferedRWPair are wrong. You should not use BufferedRWPair with the same underlying stream (that's the whole point of BufferedRWPair).

    11. pitrou commented on Aug 19, 2011

      @pitrou
      pitrou
      on Aug 19, 2011
      Member

      Here is a patch.

    12. python-dev commented on Aug 20, 2011

      @python-dev
      python-devmannequin
      on Aug 20, 2011
      Mannequin

      New changeset 5330af45f777 by Antoine Pitrou in branch '3.2':
      Issue bpo-12213: Fix a buffering bug with interleaved reads and writes that
      http://hg.python.org/cpython/rev/5330af45f777

      New changeset d7f6391954cf by Antoine Pitrou in branch 'default':
      Issue bpo-12213: Fix a buffering bug with interleaved reads and writes that
      http://hg.python.org/cpython/rev/d7f6391954cf

    13. pitrou commented on Aug 20, 2011

      @pitrou
      pitrou
      on Aug 20, 2011
      Member

      This should be fixed for BufferedRandom. As I said, I don't think BufferedRWPair is buggy.

    14. pitrou
      changed the title [-]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/-] [+]BufferedRandom: issues with interlaced read-write[/+] on Aug 20, 2011

    15. vstinner commented on Aug 20, 2011

      @vstinner
      vstinner
      on Aug 20, 2011
      MemberAuthor

      You should not use BufferedRWPair with the same underlying stream (that's the whole point of BufferedRWPair).

      It might be documented. Something like "Warning: don't use the same
      stream as reader and writer, or the BufferedRWPair becomes inconsistent
      on interlaced read-write" ?

    16. vstinner
      changed the title [-]BufferedRandom: issues with interlaced read-write[/-] [+]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/+] on Aug 20, 2011

    17. python-dev commented on Aug 20, 2011

      @python-dev
      python-devmannequin
      on Aug 20, 2011
      Mannequin

      New changeset cf2010e9f941 by Antoine Pitrou in branch '2.7':
      Issue bpo-12213: Fix a buffering bug with interleaved reads and writes that
      http://hg.python.org/cpython/rev/cf2010e9f941

    18. Arfrever
      changed the title [-]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/-] [+]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/+] on Aug 20, 2011
    19. pitrou
      changed the title [-]BufferedRandom, BufferedRWPair: issues with interlaced read-write[/-] [+]BufferedRandom: issues with interlaced read-write[/+] on Aug 20, 2011

    20. python-dev commented on Aug 20, 2011

      @python-dev
      python-devmannequin
      on Aug 20, 2011
      Mannequin

      New changeset 524896c305ce by Antoine Pitrou in branch '3.2':
      Issue bpo-12213: make it clear that BufferedRWPair shouldn't be called with the
      http://hg.python.org/cpython/rev/524896c305ce

      New changeset a423bd492d6c by Antoine Pitrou in branch 'default':
      Issue bpo-12213: make it clear that BufferedRWPair shouldn't be called with the
      http://hg.python.org/cpython/rev/a423bd492d6c

      New changeset dd4f29e39756 by Antoine Pitrou in branch '2.7':
      Issue bpo-12213: make it clear that BufferedRWPair shouldn't be called with the
      http://hg.python.org/cpython/rev/dd4f29e39756

    21. pitrou commented on Aug 20, 2011

      @pitrou
      pitrou
      on Aug 20, 2011
      Member

      I think it can be closed now.

    22. pitrou
      closed this as completedon Aug 20, 2011

    23. python-dev commented on Sep 4, 2011

      @python-dev
      python-devmannequin
      on Sep 4, 2011
      Mannequin

      New changeset 8de8945ebfc3 by Antoine Pitrou in branch '3.2':
      Issue bpo-12213: make it clear that BufferedRWPair shouldn't be called with the
      http://hg.python.org/cpython/rev/8de8945ebfc3

    24. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      topic-IOtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions