Repository navigation
IPython & PyCharm debugging mode: I/O operation on closed file #202
Description
Activity
That's a really weird error. I can't guess why there is an IO error since
tqdmstreams to the console normally. But it seemscoloramamight be at fault here.Could you please try to uninstall
coloramaand retry your script several times in order to see if the error happens again? (you will need to restart your ipython notebook kernel before to force the reloading of modules).If this still produces an IO error randomly, could you please give us the script (or better a minimal example)?
Here is code that fails for me. The function takes a list of images from two cameras and stacks them together to produce one output image.
Like the OP, I use tqdm a lot and most of the time it works wonderfully. However anything involving a loop over files seems to cause sporadic problems.
import numpy as np from tqdm import tqdm import cv2 def stack_images(l, r): lim = cv2.imread(l[0], cv2.IMREAD_GRAYSCALE) rim = cv2.imread(r[0], cv2.IMREAD_GRAYSCALE) for l_i, r_i in tqdm(zip(l[1:],r[1:])): lim_new = cv2.imread(l_i, cv2.IMREAD_GRAYSCALE) if lim_new is None: raise IOError lim = np.maximum(lim, lim_new) rim_new = cv2.imread(r_i, cv2.IMREAD_GRAYSCALE) if rim_new is None: raise IOError rim = np.maximum(rim, rim_new) return lim, rimCalled something like:
import glob left_ims = glob.glob('./left_*.png') right_ims = glob.glob('./right_*.png') left_stack, right_stack = stack_images(left_ims, right_ims)I don't get an exception if I remove the tqdm wrapper.
Hope that helps!
Just to note that I have observed the same occasional
ValueError. It is hard to reproduce. Maybe the following is helpful: python 2.7. Error occurred after approx 400K iteration of a 15 minute job.This is... frustrating. I don't want to say this, but... bug in python?
@jveitchmichaelis maybe you need an extra wait?
import numpy as np from tqdm import tqdm import cv2 import time.sleep def stack_images(r): rim = cv2.imread(r[0], cv2.IMREAD_GRAYSCALE) for r_i in tqdm(r[1:]): rim_new = cv2.imread(r_i, cv2.IMREAD_GRAYSCALE) if rim_new is None: raise IOError rim = np.maximum(rim, rim_new) time.sleep(1) # wait for possible io to fininsh? return rim # ... import glob left_ims = glob.glob('./left_*.png') right_ims = glob.glob('./right_*.png') # ensure same length left_ims = left_ims[:min(len(left_ims), len(right_ims))] right_ims = right_ims[:len(left_ims)] # stack left_stack = stack_images(left_ims) right_stack = stack_images(right_ims)
I'll try that, although I have around 30-40k images to process so it might take a while (I guess 1s was for demonstration rather than anything else). Typically this thing runs at around 40 iterations per second (i.e. 80 images/sec) and there isn't a problem when
tqdmisn't used.There was a reason I read in both images in the same function, but I forget why (I think previously I was doing a joint operation on them as I read them in) :)
My reasoning about the 1 second wait was that maybe your entire program is ending just before all of the file handles are properly closed or something. It's just a wait right at the end of the program.
Ah sorry misread that! But it fails mid-sequence, not at the end. At least I think it does, I'll check.
I have this bug too and it's frustrating. It only began very recently (within the last week); tqdm was rock-solid before that.
In a long-running task, sometimes tqdm throws an exception 30% of the way in. It doesn't just happen at the end.
(I'm using tqdm inside an IPython Notebook. Will report back with a traceback if it's separate from above.)
My traceback appears in the same place, but it also happens on Linux:
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/gcr/share/behance/mturk-tools/incremental_learning.pyc in load(mids) 36 return np.array([ 37 h5_features[mid_to_idx[mid]] ---> 38 for mid in tqdm(list(mids)) 39 ]) 40 return load /usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in __iter__(self) 620 else ncols), 621 self.desc, ascii, unit, unit_scale, --> 622 1 / avg_time if avg_time else None, bar_format)) 623 624 if self.pos: /usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in print_status(s) 96 def print_status(s): 97 len_s = len(s) ---> 98 fp_write('\r' + s + (' ' * max(last_printed_len[0] - len_s, 0))) 99 fp.flush() 100 last_printed_len[0] = len_s /usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in fp_write(s) 90 91 def fp_write(s): ---> 92 fp.write(_unicode(s)) 93 94 last_printed_len = [0] # closure over mutable variable (fast) /usr/local/lib/python2.7/dist-packages/ipykernel/iostream.pyc in write(self, string) 315 316 is_child = (not self._is_master_process()) --> 317 self._buffer.write(string) 318 if is_child: 319 # newlines imply flush in subprocesses ValueError: I/O operation on closed fileWhen you say only in the last week, do you mean you regularly update your
python packages? What was the last tqdm version which was stable for you?
When you get the error, is it always at 30% through that loop, or sometimes
earlier/later?An easy fix for now would be for me to catch the exception internally and
ignore it, but I'd still like to get to the real cause.On 27 Jul 2016 5:05 p.m., "gcr" notifications@github.com wrote:
My traceback appears in the same place, but it also happens on Linux:
ValueError Traceback (most recent call last)
/home/gcr/share/behance/mturk-tools/incremental_learning.pyc in load(mids)
36 return np.array([
37 h5_features[mid_to_idx[mid]]
---> 38 for mid in tqdm(list(mids))
39 ])
40 return load/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in iter(self)
620 else ncols),
621 self.desc, ascii, unit, unit_scale,
--> 622 1 / avg_time if avg_time else None, bar_format))
623
624 if self.pos:/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in print_status(s)
96 def print_status(s):
97 len_s = len(s)
---> 98 fp_write('\r' + s + (' ' * max(last_printed_len[0] - len_s, 0)))
99 fp.flush()
100 last_printed_len[0] = len_s/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.pyc in fp_write(s)
90
91 def fp_write(s):
---> 92 fp.write(_unicode(s))
93
94 last_printed_len = [0] # closure over mutable variable (fast)/usr/local/lib/python2.7/dist-packages/ipykernel/iostream.pyc in write(self, string)
315
316 is_child = (not self._is_master_process())
--> 317 self._buffer.write(string)
318 if is_child:
319 # newlines imply flush in subprocessesValueError: I/O operation on closed file
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#202 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AKR9mwe_zKf8RlPCLg9evL1h7DjfHRpEks5qZ4GjgaJpZM4JROPU
.Pardon. I rarely update python packages, so I don't think they changed from then until now.
However, one confusing factor could be that this week I am doing longer-running tasks. The total output produced by tqdm is much larger than it was before. That's one difference. Maybe it could be overflowing some internal buffer? (Is there a way to measure the amount of raw output that tqdm produces?)
tqdm doesn't always throw in the same place though. Sometimes the progress bar completes; sometimes it stops at 50%. It usually works.
thanks.
can anyone confirm this is still a bug using tqdm>=4.7.5?
On 27 Jul 2016 5:49 p.m., "gcr" notifications@github.com wrote:
Pardon. I rarely update python packages, so I don't think they changed
from then until now.However, one confusing factor could be that this week I am doing
longer-running tasks. The total output produced by tqdm is much larger than
it was before. That's one difference. Maybe it could be overflowing some
internal buffer? (Is there a way to measure the amount of raw output that
tqdm produces?)tqdm doesn't always throw in the same place though. Sometimes the progress
bar completes; sometimes it stops at 50%. It usually works.—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#202 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AKR9mzYVDN9UipBZ4iyW1YA5hODR1WTGks5qZ4wegaJpZM4JROPU
.aha! i'm still seeing this behavior occasionally on
4.8.1-b71261d. i didn't save a traceback (foolish me!), will post one if i see it again.12 remaining items
I have also been getting this issue on Jupyter notebook.
@lrq3000 Would it be possible if you can post a link to the corresponding bug report you mentioned in iPython notebook repo? I'd be glad to know when this issue will be resolved 👍
@samuelleach The bug report is here: ipython/ipython#9168 and the PR that will supposedly fix the issue by v4.4 here: ipython/ipykernel#123
Reacted by Sam LeachAh the ipykernel v4.4.1 was released on 2016-08-09: https://pypi.python.org/pypi/ipykernel
So we can soon try again this issue (most packages like Anaconda gets updated one month after the pypi release usually).
Thanks for the info. Anecdotally, based on the last few hours usage of tqdm, things are stable since I did
conda update ipykerneland got version 4.4.1 👍Reacted by Stephen Karl LarroqueNot a bug with tqdm, bug fixed elsewhere
At least we hope it's fixed, I did not update ipywidgets so I did not test
it myself. Anyone, please feel free to reopen this issue if the issue
arises again with the latest ipywidgets!2016-10-19 17:25 GMT+02:00 Casper da Costa-Luis notifications@github.com:
Not a bug with tqdm, bug fixed elsewhere
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#202 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABES3pfjoeT9YhitICxGZIUSnALU34dwks5q1jaGgaJpZM4JROPU
.still, it's nice to know that
tqdmappears to be more stable thatipython, hah@lrq3000 @casperdcl Hey, I am getting this error and am not using jupyter/ipython. Its quite random, in my earlier experiments tqdm was working perfectly fine, but now this error is recurring in some of the experiments.
File "train-ner.py", line 367, in train with tqdm.tqdm(total=n_train_words, leave=False) as pbar: File "/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.py", line 812, in __init__ self.sp(self.__repr__(elapsed=0)) File "/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.py", line 176, in print_status fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0))) File "/usr/local/lib/python2.7/dist-packages/tqdm/_tqdm.py", line 169, in fp_write fp.write(_unicode(s)) IOError: [Errno 32] Broken pipeDetails
Platform : Ubuntu 16.04 tqdm : 4.19.9That's realy weird, but I can reproduce this error everytime I debug a Python file like this with PyCharm:
from tqdm import tqdm if __name__ == '__main__': for _ in tqdm(range(10)): pass
When I run the same file in PyCharm it doesn't throw this exception.
MacOs 10.15.4, Python 3.8.2, tqdm 4.46.0.- changed the title
[-]I/O operation on closed file bug[/-][+]PyCharm debugging mode: I/O operation on closed file[/+]on May 16, 2020 - changed the title
[-]PyCharm debugging mode: I/O operation on closed file[/-][+]IPython & PyCharm debugging mode: I/O operation on closed file[/+]on May 16, 2020
Eighty percent of the time I use tqdm it works perfectly. The other 20% I get 'I/O operation on closed file'.
The weird thing about it is that I can use tqdm in the same script two times. The first time it doesn't work. The second time, without changing anything , it works.
I paste below the complete traceback: