Repository navigation
Add support for IMAP IDLE in imaplib #55454
Copy link
Copy link
Closed
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-emailtype-featureA feature request or enhancementA feature request or enhancement
Description
| BPO | 11245 |
|---|---|
| Nosy | @warsaw, @ericvsmith, @bitdancer, @vadmium, @mitya57, @soltysh, @ankostis, @jdek |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-02-18.20:36:10.487>
labels = ['type-feature', 'library', 'expert-email', '3.9']
title = 'Implementation of IMAP IDLE in imaplib?'
updated_at = <Date 2019-12-19.12:58:09.754>
user = 'https://bugs.python.org/ShayRojansky'bugs.python.org fields:
activity = <Date 2019-12-19.12:58:09.754>
actor = 'jdek'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2011-02-18.20:36:10.487>
creator = 'Shay.Rojansky'
dependencies = ['18921']
files = ['27400', '37555', '48790']
hgrepos = []
issue_num = 11245
keywords = ['patch']
message_count = 24.0
messages = ['128814', '129405', '129407', '171885', '171889', '172365', '172383', '202149', '202342', '202345', '233176', '235972', '236167', '245204', '245246', '245249', '245252', '245253', '245255', '245284', '246235', '246236', '293222', '358678']
nosy_count = 17.0
nosy_names = ['barry', 'pierslauder', 'eric.smith', 'piers', 'r.david.murray', 'Shay.Rojansky', 'martin.panter', 'mitya57', 'maciej.szulik', 'nafur', 'dveeden', 'Malina', 'F.Malina', 'ankostis', 'equaeghe', 'ohreally', 'jdek']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11245'
versions = ['Python 3.9']Linked PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-emailtype-featureA feature request or enhancementA feature request or enhancement
Activity
Malina commentedon Feb 18, 2015
Imaplib2 now supports Python 3. Piers and me propose to merge imaplib2 into standard library as imaplib.
Excerpt from our conversation:
Piers: ...Thanks for bringing it (this thread) to my attention. I entirely agree with your comments.
Me: ...I found the criticism of the "threads - a heavy solution"? counterproductive. Not that I know anything about threads...
Piers: I'm not sure what the whole anti-threads thing was about all those years ago since I always loved using them. Maybe early implementations were
slow, or, more likely, early adopters were clumsy ("giving threads to
a novice is like giving a blow torch to a baby" to paraphrade a quote :-)
bitdancer commentedon Jun 12, 2015
Are you volunteering to be maintainer, and/or is Piers? If he's changed his mind about the threading, that's good enough for me (and by now he has a lot more experience with the library in actual use).
The biggest barrier to inclusion, IMO, is tests and backward compatibility. There have been enough changes that making sure we don't break backward compatibility will be important, and almost certainly requires more tests than we have now. Does imaplib2 have a test suite?
We would need to get approval from python-dev, though. We have ongoing problems with packages that are maintained outside the stdlib...but updating to imaplib2 may be better than leaving it without a maintainer at all.
Can we get Piers involved in this conversation directly?
FMalina commentedon Jun 12, 2015
I am in for my part and I emailed Piers to come and join us and he surely
will when the bug tracker is responsive again.
Imaplib2 does have an up to date test suite and compatibility wise imaplib2
can be substituted for imaplib in existing clients with no changes in the
code.
On top of that I have a private IDLE test suite for common tasks such as
major email providers.
I am looking to make it part of an external project
https://github.com/fmalina/emails, but need to extract much of the recipes
first out of a working application in a reusable manner as I need it in
other projects and will do for years to come.
bitdancer commentedon Jun 12, 2015
This is great.
When you say it is fully compatible, though, is that testing against imaplib in python2 or python3? It is the python3 decisions about string/bytes handling where the discrepancies are most likely to arise, unless the python3 port was modeled on the stdlib version.
FMalina commentedon Jun 12, 2015
I just wen’t through my repo looking at relevant commits to double check and I didn’t have to change a line in my user level code when upgrading from python2 to 3. There was only one way to do it.
bitdancer commentedon Jun 12, 2015
Do you have any tests that use non-ascii passwords? I think that was the most significant bug.
FMalina commentedon Jun 12, 2015
I don’t have a test for it, neither has stdlib imaplib. We just need to port over the encode fix.
Copy over the fixed version of _CRAM_MD5_AUTH.
from line 599 in python3.5 imaplib
https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599 <https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599\>
corresponding to line 884 in imaplib2
https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884 <https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884\>
piers commentedon Jun 13, 2015
Hi, apologies for not responding to the "pierslauder" pings, but i don't own that login, or at least have forgotten all about it, and its email address is invalid (or there is another pierslauder out there).
I maintain imaplib2 on sourceforge (as piersrlauder) at https://sourceforge.net/projects/imaplib2/
and that version has just been modified to incorporate the CRAM_MD5_AUTH change from python3.6. It is regularly updated with bug fixes and it also has built-in tests for the IDLE function.
I originally intended for imaplib2 to be incorporated into pythonlib, leaving the original module in place (a la urllib/2). Then people wouldn't be forced into a switch using threads except by choice.
Anyway, happy to help.
bitdancer commentedon Jul 4, 2015
Thanks, Piers!
Sorry for dropping off the map on this, I've been busy.
I'll post to python-dev about this and see how the community would like to proceed.
bitdancer commentedon Jul 4, 2015
By the way, the pierslauder id points to 'pierslauder@users.sourceforge.net'.
ankostis commentedon May 8, 2017
Before merging
imaplib2please consider making proper use of the Python's standardloggingmodule.jdek commentedon Dec 19, 2019
Hi, I'm new to python but I had a go at implementing this for imaplib(1) using a different approach. It works but it has a couple issues (see patch), I would appreciate any thoughts/improvements.
trentbuck commentedon Nov 7, 2022
FYI,
Here is a bare-minimum version that Works For Me (so far) with python 3.9 and dovecot 1.2.9 (don't ask).
This is meant to replace a crappy cron job
* * * * * nobody curl https://⋯/receiveEmail.php >/dev/null.I'm think sure this will eventually OOM, as I never explicitly reap the "still here" continuation responses.
foresto commentedon Jul 1, 2024
@bitdancer I wonder if this is a realistic use case. Does imaplib support sending? Does the protocol? The RFCs say no:
Given that limitation inherent to the protocol, I would think it sufficient for an
idle()method to simply block, until one of these things occurs:I think repeatedly long-polling with a timeout (case 2) is a pretty common pattern in systems programming and communication web services, so probably familiar to a lot of programmers, and easy enough for calling code to implement. Clients would likely have to do this anyway, in order to avoid being disconnected by a server with an autologout timer. (The RFCs suggest a 29 minute IDLE for this reason.)
One nice thing about this approach is that it gets the job done without departing from the library's existing single-threaded design. This keeps the API semantics consistent, and avoids introducing the complexities of multi-threading into programs that use it.
I spent today working on an implementation. At roughly 50 lines of new code plus comments and doc strings, it's already functional. It adheres to the spec in areas where other attempts I've seen do not. (Notably, it avoids inventing new states or commands, avoids assumptions about what IDLE events the server will push, and avoids losing previously collected data.) It uses imaplib's existing machinery to do all the parsing and I/O, so new tests for those things would presumably not be needed.
Other than logging, the main thing left to add is the timeout. I plan to use
select()for the common cases: socket-based connections on all platforms, and stdin/stdout pipes on unix. For the special case of stdin/stdout on Windows, I plan to let the timeout be delayed until the next untagged response arrives (becauseselect()doesn't work on Windows pipes) or perhaps just let the timeout be disabled in that case. Either way, it can be documented with an OS availability note, much like various other parts of the Python stdlib.foresto commentedon Jul 15, 2024
Digging deeper into this reveals that supporting IDLE with a naïve timeout implementation is doomed to work poorly, because imaplib creates its file-like objects in buffered mode. That's a problem for any timeout based on
select()orpoll(), because those system calls can't see already-buffered data, leading them to block even when data is ready for reading.I see two possible ways to deal with this, neither of which is trivial:
read()andreadline()with custom implementations that expose their internal buffer, and check that buffer before callingselect()for an idle timeout. The file object setup code would be slightly different for the socket-based IMAP4 classes vs. IMAP4_stream, but the key methods could be shared.The risk I see with approach 1 is that, since imaplib named its file objects with no leading underscore, there might be client code in the wild that uses them directly and depends on their original (buffered) behavior. If such code exists, it would likely break.
Breakage could be avoided by subclassing IMAP4, IMAP4_SSL, and IMAP4_stream, and implementing IDLE only in the new subclasses.
Alternatively, a custom file-like class implementing all 8 read and write methods from io.BufferedReader, and exposing its internal buffer as needed for
select(), could be written and used instead of the one from the standard library. This should work even with client code that uses imaplib's file objects directly, so long as it doesn't check the types of those objects withisinstance()et al.(Then again, perhaps it's okay to break attributes like
IMAP4.fileandIMAP4_stream.readfile, given that they do not appear in the documentation?)IMAP4._get_line(), try to always completely drain the buffer on every read, and useselect()for timeouts despite having no visibility into the buffer.Note that having to drain the buffer on every read would preclude a "get one response during IDLE" method; users would have to be content with receiving multiple responses per method call, unless another buffer was added to dole them out one at a time.
The risk I see with approach 2 is that the file object comes from
socket.makefile(), which forbids nonblocking mode and strongly discourages socket timeouts, according to its documentation:That suggests to me that, although it might work in current CPython versions on popular platforms, it might eventually fail on other python implementations, versions, or operating systems. And since the internal buffer is affected, the failure could mean silent data loss or corruption.
Approach 2 is what imapclient does.
Approach 1 (using subclasses) is what I did in my own code.
foresto commentedon Jul 15, 2024
My first implementation used callbacks to handle idle responses, but it felt a little awkward, so I switched to an iterable context manager instead. Using it goes something like this:
The IDLE command is sent upon entering the context, and DONE is sent at exit. Untagged responses arriving with the server's continuation request are queued for delivery by the iterator.
The optional
durationargument limits the idle duration, for example to avoid a server-imposed inactivity timeout, or to make sure an exception is eventually raised if the network disappears during idle.To get a single response from an idle session:
Any received leftovers are appended to
untagged_responseson context exit, so they can be collected in the usual imaplib way.The iterator has a (generator) method to get the next burst of responses, such as a rapid-fire series of EXPUNGE after a bulk delete:
The burst method also works within iteration loops, and respects
duration:To spend 29 minutes processing response bursts:
...Or just use it the simple way:
foresto commentedon Jul 26, 2024
Discussion started here:
https://discuss.python.org/t/gauging-interest-in-my-imap4-idle-implementation-for-imaplib/59272
pythongh-55454: Support IMAP IDLE in imaplib
pythongh-55454: Support IMAP IDLE in imaplib
pythongh-55454: Support IMAP IDLE in imaplib
pythongh-55454: Support IMAP IDLE in imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Add IMAP4 IDLE support to imaplib
pythongh-55454: Clarify imaplib idle() docs
pythongh-55454: Clarify imaplib idle() docs
[-]Implementation of IMAP IDLE in imaplib?[/-][+]Add support for IMAP IDLE in `imaplib`[/+]serhiy-storchaka commentedon Dec 17, 2024
I am sorry, but I am not sure the interface proposed in #122542 is the best possible interface. There are some issues with it:
withblock. After client sent DONE, server can still send status updates. What happens to them? You cannot useIdlerafter leaving thewithblock. We need a method to send DONE and read status updates that came after this.untagged_responcesafter the NOOP command, although this is not thread-safe. It would be nice to have unified way to get status updates for IDLE and other commands. It would be nice if it is thread-safe or there is a clear way to use the IMAP4 client in thread-safe way.Let's look what other solutions provide. Interesting, there were two different third-party packages named
imaplib2, both were proposed for inclusion in the stdlib.Piers Lauder's
imaplib2was discussed above. It is available on PyPI. I has a simple blockingidle()method which waits for first server response or expiration of the timeout. Then it send DONE and wait for tagged response for the IDLE command. You can read status updates fromuntagged_responcesafter this. It may be not very efficient, because you need to send IDLE again, but it fits perfectly in the line with other commands. This package is compatible withimaplib. The main disadvantage of this package is that it runs internal threads. But I think that there may be possibility to implement it without running internal thread. Instead the user will need to run external threads to utilize the IMAP4 concurrency features.imaplibwill still need to use locks and events to make its methods thread-safe.Maxim Khitrov's
imaplib2was discussed in Python-ideas. It provides incompatible interface, in particularly concurrentfetch(). Its interface for IDLE is similar to the one proposed in #122542, with support for the context manager and iterator protocols, but it also has thepoll()method.Looking at support in other programming languages, in Perl, support for IDLE is provided by three methods --
idle,idle_dataanddone.idleis non-blocking, it simply sends the IDLE command,idle_datais non-blocking or blocking with timeout, it polls server response.donesends DONE.serhiy-storchaka commentedon Dec 17, 2024
I think that we first should implement support for concurrent execution of other commands (or at least draw a plan for it). Then we can add support for the IDLE command which fits the general design.
I think that many (if not all) commands should have a non-blocking variant. Either boolean parameter or a method with special suffix. Non-blocking method sends the command but does not wait for the tagged response. It should return something Future-like. You can block a thread waiting on the response for specific command or check if there is response without blocking.
There should be a thread-safe method for polling non-tagged responses (after IDLE or other commands). It should provide blocking and non-blocking interface. Iterator protocol also can be supported for convenience -- you can iterate only already received non-tagged responses or block to wait on new responses.
IDLE will also have blocking and non-blocking variants. In blocking variant it will wait on tagged server response (which can only be sent after client sent DONE in other thread), in non-blocking variant it will return a Future-like object (which also supports the context manager protocol and has the
done()method). We can also implement auto-deblocking option like in Piers Lauder'simaplib2-- send DONE after first non-tagged response and wait for tagged response.foresto commentedon Dec 20, 2024
That's not a requirement of the iterator protocol. It supports stopping immediately without producing any results, much like the read() system call can do with a non-blocking file descriptor. As a matter of fact, #122542 already does this when a limited-duration Idler has expired.
It does block in other cases, and this is by design, because the whole of imaplib is a blocking API. #122542 respects and preserves that design. It's a small extension to the existing module, not a replacement for it.
(And just to be clear for the sake of other readers, the IDLE command doesn't poll, at least not in the way we usually mean in computer science. The point of IDLE is to avoid polling, and instead react to events sent by the server as they arrive.)
There are two ways. Waiting can be cleanly interrupted by:
If you're wishing for an API to interrupt a blocking IDLE from another thread, then it is true that no such thing exists at the moment. The same is true for every other command in imaplib.
Such a thing could be added in the future. One way would be modifying imaplib to use non-bocking sockets, and make blocking commands call select() on both the IMAP socket and one end of a socket pair. Another thread could then write to the other end of the socket pair to interrupt the command. The interface in #122542 would not have to change.
Of course, moving imaplib to non-blocking sockets and adding calls to interrupt its blocking commands would be additional features, beyond the scope of #122542. This PR simply adds support for IMAP IDLE. It doesn't aim to overhaul or replace imaplib.
Let's keep in mind that introducing IDLE support would't mean all applications have to use it. They can continue using the existing commands, and poll for updates using NOOP, just as they always have.
When considering applications with asynchronous display updates, let's also remember that all of imaplib is already a blocking API, so the issues around blocking calls are already present. An application can deal with them by running commands in a separate thread. #122542 doesn't change this. The same approach can be used with IDLE.
Note that IMAP doesn't allow other commands during IDLE, so an application wishing to do both at the same time would have to keep two connections open, presumably in separate threads. That's a limitation imposed by the protocol.
(I suppose a library could hide this limitation by automatically switching between command mode and IDLE mode, making the switches transparent to application code, but such a library would be considerably more complex than imaplib.)
This doesn't exist in any of imaplib's commands. I agree that it could be useful for certain programming models, but it's a separate concern from adding IDLE support. I hope we can agree that the two should not be conflated.
Are you worried that #122542 might get in the way of future non-blocking support in imaplib? I don't believe it would.
Indeed, the server can send untagged responses after DONE is sent. These are already handled in #122542. They can be retrieved with the response() method, just as they can after other commands.
Both can produce untagged responses, but their similarity ends there. Notably:
IDLE and NOOP are made for different situations, and their interfaces reflect that.
The untagged_responses attribute is an internal, undocumented interface. Programs should not be using it. The correct way to do this is through the response() method. This still works in #122542, including responses after leaving the IDLE context (as mentioned above).
This could be done with a callback interface, iteration, or polling. (Although using a polling interface for IDLE would be a bit silly, because its purpose is to avoid polling; that's what NOOP is for.)
However, it would mean modifying or duplicating every command method in imaplib, not just IDLE, and adding features that are orthogonal to the present issue. I think that would make more sense as a separate feature request. Don't you?
I guess you mean sharing a single IMAP4 object and connection among multiple threads. (As far as I know, imaplib is already thread-safe when each thread uses its own IMAP4 object. I don't see any writes to global variables, at least.)
I think this would be a little tricky even without IDLE, since you would have to multiplex commands from different threads into a single connection, ensure atomic writes to the server, sort out the various tagged and untagged responses, and route them to the threads that wanted them. It would be even harder with IDLE, because IMAP doesn't support command mode and IDLE mode at the same time. You would have to dynamically switch between modes within the library while simulating a single mode from each thread's point of view.
I suppose that might be possible, but would be a major undertaking, and the result would arguably not be imaplib any more. If it's important to you, I think you should write up a separate issue for it. This one is about adding IDLE support to imaplib.
I've seen that one. #122542 already does all those things, and more, and it does them more elegantly, and without imposing threads on applications.
I've seen that one, and no, its interface for IDLE is not similar to #122542. Note that its iteration feature doesn't work during IDLE. Instead, it forces the application to use a polling loop and make select() or sleep() calls in between. It also has various implementation flaws that will make it misbehave in conditions where #122542 works correctly. Perhaps those could be fixed, but there's a bigger problem with it:
The interface is incompatible with imaplib, as you pointed out. So adopting it would effectively mean replacing imaplib.
My takeaway from everything you've written here is that you would like a non-blocking IMAP client library with polling semantics.
I can see ways in which that would be useful, but imaplib is not that library, and there is no IDLE design or implementation that can change that. The imaplib API is a blocking design throughout. Getting what you want would require substantial changes all across the library, or else replacing it.
You're not alone in your wish. An imaplib overhaul or replacement has been suggested every so often for at least 13 years. However, nobody has done the work to make it happen in the standard library, and there is no sign that anyone will have done that work any time soon.
Meanwhile, we finally have a clean, working, compatible, pythonic, documented, reviewed pull request providing IDLE support, which people have also been requesting for at least 13 years. And as far as I can tell, there is nothing about it that would prevent the non-blocking interface you want from being added later.
I therefore propose that the request for non-blocking imaplib interfaces (or replacing imaplib entirely) be moved to a new issue.
gh-55454: Add IMAP4 IDLE support to imaplib (#122542)
gvanrossum commentedon Feb 7, 2025
This landed, so closing.
pythongh-55454: Add IMAP4 IDLE support to imaplib (python#122542)
pythongh-55454: Add IMAP4 IDLE support to imaplib (python#122542)