Skip to content

Add support for IMAP IDLE in imaplib #55454

Closed

Description

@ShayRojansky
BPO 11245
Nosy @warsaw, @ericvsmith, @bitdancer, @vadmium, @mitya57, @soltysh, @ankostis, @jdek
Dependencies
  • bpo-18921: In imaplib, cached capabilities may be out of date after login
  • Files
  • imapidle.patch
  • imapidle.patch: New patch
  • 0001-Add-IDLE-support-to-imaplib.patch: IMAP IDLE patch for imaplib
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2011-02-18.20:36:10.487>
    labels = ['type-feature', 'library', 'expert-email', '3.9']
    title = 'Implementation of IMAP IDLE in imaplib?'
    updated_at = <Date 2019-12-19.12:58:09.754>
    user = 'https://bugs.python.org/ShayRojansky'

    bugs.python.org fields:

    activity = <Date 2019-12-19.12:58:09.754>
    actor = 'jdek'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)', 'email']
    creation = <Date 2011-02-18.20:36:10.487>
    creator = 'Shay.Rojansky'
    dependencies = ['18921']
    files = ['27400', '37555', '48790']
    hgrepos = []
    issue_num = 11245
    keywords = ['patch']
    message_count = 24.0
    messages = ['128814', '129405', '129407', '171885', '171889', '172365', '172383', '202149', '202342', '202345', '233176', '235972', '236167', '245204', '245246', '245249', '245252', '245253', '245255', '245284', '246235', '246236', '293222', '358678']
    nosy_count = 17.0
    nosy_names = ['barry', 'pierslauder', 'eric.smith', 'piers', 'r.david.murray', 'Shay.Rojansky', 'martin.panter', 'mitya57', 'maciej.szulik', 'nafur', 'dveeden', 'Malina', 'F.Malina', 'ankostis', 'equaeghe', 'ohreally', 'jdek']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'test needed'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue11245'
    versions = ['Python 3.9']

    Linked PRs

    Activity

    1. Malina commented on Feb 18, 2015

      Malinamannequin
      Mannequin

      Imaplib2 now supports Python 3. Piers and me propose to merge imaplib2 into standard library as imaplib.

      Excerpt from our conversation:

      Piers: ...Thanks for bringing it (this thread) to my attention. I entirely agree with your comments.

      Me: ...I found the criticism of the "threads - a heavy solution"? counterproductive. Not that I know anything about threads...

      Piers: I'm not sure what the whole anti-threads thing was about all those years ago since I always loved using them. Maybe early implementations were
      slow, or, more likely, early adopters were clumsy ("giving threads to
      a novice is like giving a blow torch to a baby" to paraphrade a quote :-)

    2. bitdancer commented on Jun 12, 2015

      @bitdancer
      Member

      Are you volunteering to be maintainer, and/or is Piers? If he's changed his mind about the threading, that's good enough for me (and by now he has a lot more experience with the library in actual use).

      The biggest barrier to inclusion, IMO, is tests and backward compatibility. There have been enough changes that making sure we don't break backward compatibility will be important, and almost certainly requires more tests than we have now. Does imaplib2 have a test suite?

      We would need to get approval from python-dev, though. We have ongoing problems with packages that are maintained outside the stdlib...but updating to imaplib2 may be better than leaving it without a maintainer at all.

      Can we get Piers involved in this conversation directly?

    3. FMalina commented on Jun 12, 2015

      FMalinamannequin
      Mannequin

      I am in for my part and I emailed Piers to come and join us and he surely
      will when the bug tracker is responsive again.

      Imaplib2 does have an up to date test suite and compatibility wise imaplib2
      can be substituted for imaplib in existing clients with no changes in the
      code.

      On top of that I have a private IDLE test suite for common tasks such as

      • instant bounce processing,
      • email photo upload to web service,
      • unsubscribe processing via list-unsubscribe headers and feedback loops of
        major email providers.

      I am looking to make it part of an external project
      https://github.com/fmalina/emails, but need to extract much of the recipes
      first out of a working application in a reusable manner as I need it in
      other projects and will do for years to come.

    4. bitdancer commented on Jun 12, 2015

      @bitdancer
      Member

      This is great.

      When you say it is fully compatible, though, is that testing against imaplib in python2 or python3? It is the python3 decisions about string/bytes handling where the discrepancies are most likely to arise, unless the python3 port was modeled on the stdlib version.

    5. FMalina commented on Jun 12, 2015

      FMalinamannequin
      Mannequin

      I just wen’t through my repo looking at relevant commits to double check and I didn’t have to change a line in my user level code when upgrading from python2 to 3. There was only one way to do it.

    6. bitdancer commented on Jun 12, 2015

      @bitdancer
      Member

      Do you have any tests that use non-ascii passwords? I think that was the most significant bug.

    7. FMalina commented on Jun 12, 2015

      FMalinamannequin
      Mannequin

      I don’t have a test for it, neither has stdlib imaplib. We just need to port over the encode fix.

      Copy over the fixed version of _CRAM_MD5_AUTH.
      from line 599 in python3.5 imaplib
      https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599 <https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599\>
      corresponding to line 884 in imaplib2
      https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884 <https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884\>

    8. piers commented on Jun 13, 2015

      piersmannequin
      Mannequin

      Hi, apologies for not responding to the "pierslauder" pings, but i don't own that login, or at least have forgotten all about it, and its email address is invalid (or there is another pierslauder out there).

      I maintain imaplib2 on sourceforge (as piersrlauder) at https://sourceforge.net/projects/imaplib2/
      and that version has just been modified to incorporate the CRAM_MD5_AUTH change from python3.6. It is regularly updated with bug fixes and it also has built-in tests for the IDLE function.

      I originally intended for imaplib2 to be incorporated into pythonlib, leaving the original module in place (a la urllib/2). Then people wouldn't be forced into a switch using threads except by choice.

      Anyway, happy to help.

    9. bitdancer commented on Jul 4, 2015

      @bitdancer
      Member

      Thanks, Piers!

      Sorry for dropping off the map on this, I've been busy.

      I'll post to python-dev about this and see how the community would like to proceed.

    10. bitdancer commented on Jul 4, 2015

      @bitdancer
      Member

      By the way, the pierslauder id points to 'pierslauder@users.sourceforge.net'.

    11. ankostis commented on May 8, 2017

      ankostismannequin
      Mannequin

      Before merging imaplib2 please consider making proper use of the Python's standard logging module.

    12. jdek commented on Dec 19, 2019

      jdekmannequin
      Mannequin

      Hi, I'm new to python but I had a go at implementing this for imaplib(1) using a different approach. It works but it has a couple issues (see patch), I would appreciate any thoughts/improvements.

    13. transferred this issue fromon Apr 10, 2022
    14. trentbuck commented on Nov 7, 2022

      @trentbuck

      FYI,

      Here is a bare-minimum version that Works For Me (so far) with python 3.9 and dovecot 1.2.9 (don't ask).
      This is meant to replace a crappy cron job * * * * * nobody curl https://⋯/receiveEmail.php >/dev/null.
      I'm think sure this will eventually OOM, as I never explicitly reap the "still here" continuation responses.

      #!/usr/bin/python3 
      
      import imaplib
      import logging
      
      
      # Enable IDLE support. 
      if 'IDLE' in imaplib.Commands:
      
          # in the unlikely event this feature is fixed upstream... 
          IMAP4_SSL = imaplib.IMAP4_SSL
      
      else:
      
          class IMAP4_SSL_plus_IDLE(imaplib.IMAP4_SSL):
      
              def idle(self):
                  if 'IDLE' not in self.capabilities:
                      raise self.error('Server does not support IDLE')
                  idle_tag = self._command('IDLE')  # start idling 
                  self._get_response()
                  while line := self._get_line():
                      if line.endswith(b'EXISTS'):
                          self.send(b'DONE' + imaplib.CRLF)
                          return self._command_complete('IDLE', idle_tag)
      
          imaplib.Commands['IDLE'] = ('AUTH', 'SELECTED')
          IMAP4_SSL = IMAP4_SSL_plus_IDLE
      
      
      with IMAP4_SSL(⋯) as conn:
          conn.login(user=⋯, password=⋯)
          conn.select(mailbox='INBOX', readonly=True)
          conn.debug = 100            # DEBUGGING 
          while True:
              resp = requests.get('https://⋯/receiveEmail.php')
              resp.raise_for_status()
              logging.debug('HTTP GET said %s', resp.text)
              resp = conn.idle()
              logging.debug('IMAP IDLE said %s', resp)
      
    15. foresto commented on Jul 1, 2024

      @foresto
      Contributor

      Looking at this again, it appears as though there's no way to interrupt IDLE if you want to, say, send an email.

      @bitdancer I wonder if this is a realistic use case. Does imaplib support sending? Does the protocol? The RFCs say no:

      rfc3501: "IMAP4rev1 does not specify a means of posting mail; this function is handled by a mail transfer protocol such as RFC 2821."

      rfc9051: "IMAP4rev2 does not specify a means of posting mail; this function is handled by a mail submission protocol such as the one specified in RFC 6409."

      Given that limitation inherent to the protocol, I would think it sufficient for an idle() method to simply block, until one of these things occurs:

      1. A callback for receiving untagged data says "that's enough; we're done idling."
      2. A caller-supplied idle timeout expires.

      I think repeatedly long-polling with a timeout (case 2) is a pretty common pattern in systems programming and communication web services, so probably familiar to a lot of programmers, and easy enough for calling code to implement. Clients would likely have to do this anyway, in order to avoid being disconnected by a server with an autologout timer. (The RFCs suggest a 29 minute IDLE for this reason.)

      One nice thing about this approach is that it gets the job done without departing from the library's existing single-threaded design. This keeps the API semantics consistent, and avoids introducing the complexities of multi-threading into programs that use it.

      I spent today working on an implementation. At roughly 50 lines of new code plus comments and doc strings, it's already functional. It adheres to the spec in areas where other attempts I've seen do not. (Notably, it avoids inventing new states or commands, avoids assumptions about what IDLE events the server will push, and avoids losing previously collected data.) It uses imaplib's existing machinery to do all the parsing and I/O, so new tests for those things would presumably not be needed.

      Other than logging, the main thing left to add is the timeout. I plan to use select() for the common cases: socket-based connections on all platforms, and stdin/stdout pipes on unix. For the special case of stdin/stdout on Windows, I plan to let the timeout be delayed until the next untagged response arrives (because select() doesn't work on Windows pipes) or perhaps just let the timeout be disabled in that case. Either way, it can be documented with an OS availability note, much like various other parts of the Python stdlib.

    16. foresto commented on Jul 15, 2024

      @foresto
      Contributor

      Digging deeper into this reveals that supporting IDLE with a naïve timeout implementation is doomed to work poorly, because imaplib creates its file-like objects in buffered mode. That's a problem for any timeout based on select() or poll(), because those system calls can't see already-buffered data, leading them to block even when data is ready for reading.

      I see two possible ways to deal with this, neither of which is trivial:


      1. Stop using the standard library's buffered file objects, override read() and readline() with custom implementations that expose their internal buffer, and check that buffer before calling select() for an idle timeout. The file object setup code would be slightly different for the socket-based IMAP4 classes vs. IMAP4_stream, but the key methods could be shared.

      The risk I see with approach 1 is that, since imaplib named its file objects with no leading underscore, there might be client code in the wild that uses them directly and depends on their original (buffered) behavior. If such code exists, it would likely break.

      Breakage could be avoided by subclassing IMAP4, IMAP4_SSL, and IMAP4_stream, and implementing IDLE only in the new subclasses.

      Alternatively, a custom file-like class implementing all 8 read and write methods from io.BufferedReader, and exposing its internal buffer as needed for select(), could be written and used instead of the one from the standard library. This should work even with client code that uses imaplib's file objects directly, so long as it doesn't check the types of those objects with isinstance() et al.

      (Then again, perhaps it's okay to break attributes like IMAP4.file and IMAP4_stream.readfile, given that they do not appear in the documentation?)


      1. Put the file object in nonblocking mode and/or set a socket timeout during IDLE. Handle the resulting exceptions from IMAP4._get_line(), try to always completely drain the buffer on every read, and use select() for timeouts despite having no visibility into the buffer.

      Note that having to drain the buffer on every read would preclude a "get one response during IDLE" method; users would have to be content with receiving multiple responses per method call, unless another buffer was added to dole them out one at a time.

      The risk I see with approach 2 is that the file object comes from socket.makefile(), which forbids nonblocking mode and strongly discourages socket timeouts, according to its documentation:

      "The socket must be in blocking mode; it can have a timeout, but the file object’s internal buffer may end up in an inconsistent state if a timeout occurs."

      That suggests to me that, although it might work in current CPython versions on popular platforms, it might eventually fail on other python implementations, versions, or operating systems. And since the internal buffer is affected, the failure could mean silent data loss or corruption.


      Approach 2 is what imapclient does.

      Approach 1 (using subclasses) is what I did in my own code.

    17. foresto commented on Jul 15, 2024

      @foresto
      Contributor

      My first implementation used callbacks to handle idle responses, but it felt a little awkward, so I switched to an iterable context manager instead. Using it goes something like this:

      with imap.idle(duration=29*60) as idler:
          for response in idler:
              typ, data = response
              print(typ, data)

      The IDLE command is sent upon entering the context, and DONE is sent at exit. Untagged responses arriving with the server's continuation request are queued for delivery by the iterator.

      The optional duration argument limits the idle duration, for example to avoid a server-imposed inactivity timeout, or to make sure an exception is eventually raised if the network disappears during idle.

      To get a single response from an idle session:

      with imap.idle() as idler:
          typ, data = next(idler)

      Any received leftovers are appended to untagged_responses on context exit, so they can be collected in the usual imaplib way.

      The iterator has a (generator) method to get the next burst of responses, such as a rapid-fire series of EXPUNGE after a bulk delete:

      with imap.idle() as idler:
      
          # get the next response and any others following by < 0.1 second
          batch = list(idler.burst(interval=0.1))
      
          print(f'processing {len(batch)} responses...')
          for typ, data in batch:
              print(typ, data)

      The burst method also works within iteration loops, and respects duration:

      with imap.idle(duration=29*60) as idler:
      
          for typ, data in idler:
      
              print('got a single response:', typ, data)
      
              batch = list(idler.burst())
              print(f'also got a burst of {len(batch)} more responses')

      To spend 29 minutes processing response bursts:

      with imap.idle(duration=29*60) as idler:
      
          while batch := list(idler.burst()):
      
              print(f'processing {len(batch)} responses...')
              for typ, data in batch:
                  print(typ, data)

      ...Or just use it the simple way:

      with imap.idle() as idler:
          for response in idler:
              print(response)
    18. foresto commented on Jul 26, 2024

      @foresto
      Contributor
    19. added 14 commits that reference this issue on Jul 31, 2024
    20. added a commit that references this issue on Aug 16, 2024
    21. added a commit that references this issue on Sep 1, 2024
    22. added 2 commits that reference this issue on Sep 11, 2024
    23. changed the title [-]Implementation of IMAP IDLE in imaplib?[/-] [+]Add support for IMAP IDLE in `imaplib`[/+] on Dec 3, 2024
    24. serhiy-storchaka commented on Dec 17, 2024

      @serhiy-storchaka
      Member

      I am sorry, but I am not sure the interface proposed in #122542 is the best possible interface. There are some issues with it:

      • Using the iterator protocol means that the polling is blocking. When we are blocked on waiting a response from the server, there is no way to interrupt waiting. It means that an interactive application which displays an IMAP4 folder status at runtime cannot be responsible to user input. We need a non-blocking operation to poll a response if there is one and return immediately if there are no responses. We need also a blocking operation with timeout and a way to interrupt thread before expiration of the timeout.
      • Using the context manager protocol means that DONE is only sent after leaving the with block. After client sent DONE, server can still send status updates. What happens to them? You cannot use Idler after leaving the with block. We need a method to send DONE and read status updates that came after this.
      • Server can send status updates not only for IDLE command, but for other commands. It is common to use the NOOP command for this. There is a large similarity between NOOP and IDLE commands, but is is not seen in the proposed interface. You can read status updates from untagged_responces after the NOOP command, although this is not thread-safe. It would be nice to have unified way to get status updates for IDLE and other commands. It would be nice if it is thread-safe or there is a clear way to use the IMAP4 client in thread-safe way.

      Let's look what other solutions provide. Interesting, there were two different third-party packages named imaplib2, both were proposed for inclusion in the stdlib.

      Piers Lauder's imaplib2 was discussed above. It is available on PyPI. I has a simple blocking idle() method which waits for first server response or expiration of the timeout. Then it send DONE and wait for tagged response for the IDLE command. You can read status updates from untagged_responces after this. It may be not very efficient, because you need to send IDLE again, but it fits perfectly in the line with other commands. This package is compatible with imaplib. The main disadvantage of this package is that it runs internal threads. But I think that there may be possibility to implement it without running internal thread. Instead the user will need to run external threads to utilize the IMAP4 concurrency features. imaplib will still need to use locks and events to make its methods thread-safe.

      Maxim Khitrov's imaplib2 was discussed in Python-ideas. It provides incompatible interface, in particularly concurrent fetch(). Its interface for IDLE is similar to the one proposed in #122542, with support for the context manager and iterator protocols, but it also has the poll() method.

      Looking at support in other programming languages, in Perl, support for IDLE is provided by three methods -- idle, idle_data and done. idle is non-blocking, it simply sends the IDLE command, idle_data is non-blocking or blocking with timeout, it polls server response. done sends DONE.

    25. serhiy-storchaka commented on Dec 17, 2024

      @serhiy-storchaka
      Member

      I think that we first should implement support for concurrent execution of other commands (or at least draw a plan for it). Then we can add support for the IDLE command which fits the general design.

      I think that many (if not all) commands should have a non-blocking variant. Either boolean parameter or a method with special suffix. Non-blocking method sends the command but does not wait for the tagged response. It should return something Future-like. You can block a thread waiting on the response for specific command or check if there is response without blocking.

      There should be a thread-safe method for polling non-tagged responses (after IDLE or other commands). It should provide blocking and non-blocking interface. Iterator protocol also can be supported for convenience -- you can iterate only already received non-tagged responses or block to wait on new responses.

      IDLE will also have blocking and non-blocking variants. In blocking variant it will wait on tagged server response (which can only be sent after client sent DONE in other thread), in non-blocking variant it will return a Future-like object (which also supports the context manager protocol and has the done() method). We can also implement auto-deblocking option like in Piers Lauder's imaplib2 -- send DONE after first non-tagged response and wait for tagged response.

    26. foresto commented on Dec 20, 2024

      @foresto
      Contributor

      I am sorry, but I am not sure the interface proposed in #122542 is the best possible interface. There are some issues with it:

      • Using the iterator protocol means that the polling is blocking.

      That's not a requirement of the iterator protocol. It supports stopping immediately without producing any results, much like the read() system call can do with a non-blocking file descriptor. As a matter of fact, #122542 already does this when a limited-duration Idler has expired.

      It does block in other cases, and this is by design, because the whole of imaplib is a blocking API. #122542 respects and preserves that design. It's a small extension to the existing module, not a replacement for it.

      (And just to be clear for the sake of other readers, the IDLE command doesn't poll, at least not in the way we usually mean in computer science. The point of IDLE is to avoid polling, and instead react to events sent by the server as they arrive.)

      When we are blocked on waiting a response from the server, there is no way to interrupt waiting.

      There are two ways. Waiting can be cleanly interrupted by:

      • A duration (time limit) passed to the call that sets up the IDLE context.
      • An exception. For example, a KeyboardInterrupt in an interactive terminal program, or some other exception raised by a timer or signal handler.

      If you're wishing for an API to interrupt a blocking IDLE from another thread, then it is true that no such thing exists at the moment. The same is true for every other command in imaplib.

      Such a thing could be added in the future. One way would be modifying imaplib to use non-bocking sockets, and make blocking commands call select() on both the IMAP socket and one end of a socket pair. Another thread could then write to the other end of the socket pair to interrupt the command. The interface in #122542 would not have to change.

      Of course, moving imaplib to non-blocking sockets and adding calls to interrupt its blocking commands would be additional features, beyond the scope of #122542. This PR simply adds support for IMAP IDLE. It doesn't aim to overhaul or replace imaplib.

      It means that an interactive application which displays an IMAP4 folder status at runtime cannot be responsible to user input.

      Let's keep in mind that introducing IDLE support would't mean all applications have to use it. They can continue using the existing commands, and poll for updates using NOOP, just as they always have.

      When considering applications with asynchronous display updates, let's also remember that all of imaplib is already a blocking API, so the issues around blocking calls are already present. An application can deal with them by running commands in a separate thread. #122542 doesn't change this. The same approach can be used with IDLE.

      Note that IMAP doesn't allow other commands during IDLE, so an application wishing to do both at the same time would have to keep two connections open, presumably in separate threads. That's a limitation imposed by the protocol.

      (I suppose a library could hide this limitation by automatically switching between command mode and IDLE mode, making the switches transparent to application code, but such a library would be considerably more complex than imaplib.)

      We need a non-blocking operation to poll a response if there is one and return immediately if there are no responses.

      This doesn't exist in any of imaplib's commands. I agree that it could be useful for certain programming models, but it's a separate concern from adding IDLE support. I hope we can agree that the two should not be conflated.

      Are you worried that #122542 might get in the way of future non-blocking support in imaplib? I don't believe it would.

      • Using the context manager protocol means that DONE is only sent after leaving the with block. After client sent DONE, server can still send status updates. What happens to them? You cannot use Idler after leaving the with block. We need a method to send DONE and read status updates that came after this.

      Indeed, the server can send untagged responses after DONE is sent. These are already handled in #122542. They can be retrieved with the response() method, just as they can after other commands.

      • Server can send status updates not only for IDLE command, but for other commands. It is common to use the NOOP command for this. There is a large similarity between NOOP and IDLE commands, but is is not seen in the proposed interface.

      Both can produce untagged responses, but their similarity ends there. Notably:

      • The IDLE command requests a temporary inversion of control, which is represented nicely in the proposed interface using a python context manager.
      • The IDLE command produces an unbounded stream of responses with unpredictable timing, which is represented nicely in the proposed interface using python iteration.

      IDLE and NOOP are made for different situations, and their interfaces reflect that.

      You can read status updates from untagged_responces after the NOOP command,

      The untagged_responses attribute is an internal, undocumented interface. Programs should not be using it. The correct way to do this is through the response() method. This still works in #122542, including responses after leaving the IDLE context (as mentioned above).

      It would be nice to have unified way to get status updates for IDLE and other commands.

      This could be done with a callback interface, iteration, or polling. (Although using a polling interface for IDLE would be a bit silly, because its purpose is to avoid polling; that's what NOOP is for.)

      However, it would mean modifying or duplicating every command method in imaplib, not just IDLE, and adding features that are orthogonal to the present issue. I think that would make more sense as a separate feature request. Don't you?

      It would be nice if it is thread-safe or there is a clear way to use the IMAP4 client in thread-safe way.

      I guess you mean sharing a single IMAP4 object and connection among multiple threads. (As far as I know, imaplib is already thread-safe when each thread uses its own IMAP4 object. I don't see any writes to global variables, at least.)

      I think this would be a little tricky even without IDLE, since you would have to multiplex commands from different threads into a single connection, ensure atomic writes to the server, sort out the various tagged and untagged responses, and route them to the threads that wanted them. It would be even harder with IDLE, because IMAP doesn't support command mode and IDLE mode at the same time. You would have to dynamically switch between modes within the library while simulating a single mode from each thread's point of view.

      I suppose that might be possible, but would be a major undertaking, and the result would arguably not be imaplib any more. If it's important to you, I think you should write up a separate issue for it. This one is about adding IDLE support to imaplib.

      Let's look what other solutions provide. Interesting, there were two different third-party packages named imaplib2, both were proposed for inclusion in the stdlib.

      Piers Lauder's imaplib2 was discussed above. It is available on PyPI. I has a simple blocking idle() method which waits for first server response or expiration of the timeout. Then it send DONE and wait for tagged response for the IDLE command. You can read status updates from untagged_responces after this. It may be not very efficient, because you need to send IDLE again, but it fits perfectly in the line with other commands. This package is compatible with imaplib. The main disadvantage of this package is that it runs internal threads. But I think that there may be possibility to implement it without running internal thread. Instead the user will need to run external threads to utilize the IMAP4 concurrency features. imaplib will still need to use locks and events to make its methods thread-safe.

      I've seen that one. #122542 already does all those things, and more, and it does them more elegantly, and without imposing threads on applications.

      Maxim Khitrov's imaplib2 was discussed in Python-ideas. It provides incompatible interface, in particularly concurrent fetch(). Its interface for IDLE is similar to the one proposed in #122542, with support for the context manager and iterator protocols, but it also has the poll() method.

      I've seen that one, and no, its interface for IDLE is not similar to #122542. Note that its iteration feature doesn't work during IDLE. Instead, it forces the application to use a polling loop and make select() or sleep() calls in between. It also has various implementation flaws that will make it misbehave in conditions where #122542 works correctly. Perhaps those could be fixed, but there's a bigger problem with it:

      The interface is incompatible with imaplib, as you pointed out. So adopting it would effectively mean replacing imaplib.

      Looking at support in other programming languages, in Perl, support for IDLE is provided by three methods -- idle, idle_data and done. idle is non-blocking, it simply sends the IDLE command, idle_data is non-blocking or blocking with timeout, it polls server response. done sends DONE.

      My takeaway from everything you've written here is that you would like a non-blocking IMAP client library with polling semantics.

      I can see ways in which that would be useful, but imaplib is not that library, and there is no IDLE design or implementation that can change that. The imaplib API is a blocking design throughout. Getting what you want would require substantial changes all across the library, or else replacing it.

      You're not alone in your wish. An imaplib overhaul or replacement has been suggested every so often for at least 13 years. However, nobody has done the work to make it happen in the standard library, and there is no sign that anyone will have done that work any time soon.

      Meanwhile, we finally have a clean, working, compatible, pythonic, documented, reviewed pull request providing IDLE support, which people have also been requesting for at least 13 years. And as far as I can tell, there is nothing about it that would prevent the non-blocking interface you want from being added later.

      I therefore propose that the request for non-blocking imaplib interfaces (or replacing imaplib entirely) be moved to a new issue.

    27. added a commit that references this issue on Feb 7, 2025
    28. gvanrossum commented on Feb 7, 2025

      @gvanrossum
      Member

      This landed, so closing.

    29. added a commit that references this issue on Feb 7, 2025
    30. added a commit that references this issue on Feb 8, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytopic-emailtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions