Repository navigation
Dataloader with multiprocessing_context="spawn" fails with multiprocessing.ValueΒ #105514
Description
π Describe the bug
If you pass a multiprocessing.Value to the dataloader workers and access it from both the main process and the workers, this results in a segmentation fault.
Code to repro:
import torch
import multiprocessing
from functools import partial
ITERS = 10
class MyDataset(torch.utils.data.Dataset):
def __getitem__(self, idx):
print(f"value: {self.mp_value.value}")
return idx
def __len__(self):
return ITERS
def worker_init(mp_value, worker_id):
worker_info = torch.utils.data.get_worker_info()
dataset = worker_info.dataset
dataset.mp_value = mp_value
if __name__ == '__main__':
ds = MyDataset()
mp_value = multiprocessing.Value("i", -1)
loader = torch.utils.data.DataLoader(ds, num_workers=16, worker_init_fn=partial(worker_init, mp_value), multiprocessing_context=multiprocessing.get_context("spawn"))
for data in loader:
mp_value.value = data
Output:
ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
File "torch/utils/data/dataloader.py", line 1132, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "multiprocessing/queues.py", line 107, in get
if not self._poll(timeout):
File "multiprocessing/connection.py", line 257, in poll
return self._poll(timeout)
File "multiprocessing/connection.py", line 424, in _poll
r = wait([self], timeout)
File "multiprocessing/connection.py", line 931, in wait
ready = selector.select(timeout)
File "selectors.py", line 415, in select
fd_event_list = self._selector.poll(timeout)
File "torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 954083) is killed by signal: Segmentation fault.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/testdataset.py", line 25, in <module>
for data in loader:
File "torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "torch/utils/data/dataloader.py", line 1328, in _next_data
idx, data = self._get_data()
File "torch/utils/data/dataloader.py", line 1294, in _get_data
success, data = self._try_get_data()
File "torch/utils/data/dataloader.py", line 1145, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 954083) exited unexpectedly
However, similar code that just uses torch.multiprocessing.spawn without the Dataloader seems to work fine if multiprocessing.Value is passed in. Seems like this is a problem with Dataloader + multiprocessing spawn. Dataloader with multiprocessing fork works fine for this example.
Versions
Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] mypy-protobuf==3.3.0
[pip3] numpy==1.23.5
[pip3] pytorch-triton==2.1.0+46672772b4
[pip3] pytorch3d==0.6.2
[pip3] torch==2.0.1+1686355048.cuda120.cudnn891.nccl218.ap
[pip3] torch-mlir==1689007104-8a8e1baf
[pip3] torch-scatter==2.1.0
[pip3] torch-tb-profiler==0.4.1
[pip3] torchfile==0.1.0
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.15.2a0+fa99a53
[pip3] triton==2.1.0
[conda] magma-cuda121 2.6.1 1 pytorch
[conda] mkl 2020.4 h726a3e6_304 conda-forge
[conda] mkl-include 2023.1.0 h84fe81f_48680 conda-forge
[conda] numpy 1.23.5 py38h7042d01_0 conda-forge
[conda] pytorch3d 0.6.2 pypi_0 pypi
[conda] torch 2.0.1+1687647636.cuda120.cudnn891.nccl218.ap pypi_0 pypi
[conda] torch-mlir 1689007104-8a8e1baf pypi_0 pypi
[conda] torch-scatter 2.1.0 pypi_0 pypi
[conda] torch-tb-profiler 0.4.1 pypi_0 pypi
[conda] torchfile 0.1.0 pypi_0 pypi
[conda] torchmetrics 0.11.4 pyhd8ed1ab_0 conda-forge
[conda] torchvision 0.15.2a0+fa99a53 pypi_0 pypi
[conda] triton 2.1.0 pypi_0 pypi
Activity
albanD commentedon Jul 19, 2023
This seems to be very related to #29855
The c++ stack trace is exactly the same:
And the python stack is similar
ezhang887 commentedon Jul 19, 2023
This diff fixes the issue for me:
This is what I think is happening (I'm using
Python 3.8, not sure if this applies to different python versions):SemLock's (a named semaphore used internally inValue) internal behavior changes a bit depending on the multiprocessing start method.Valueisforkit sets thisunlink_nowto True: link.unlink_nowisTrue, then thenameof the named semaphore's name will beNULLandsem_unlinkis called immediately: link. This leads to not being able to re-construct the named semaphore correctly when un-pickling. But this only happens if the context of theValueis fork so there's no pickling then so that's OK.mp.Valueis created with aspawncontext.ezhang887 commentedon Jul 19, 2023
@albanD Looks like the following diff might also fix #29855? It seems to fix the example there for me (I'm using
Python 3.8, not sure if the same applies to different python versions)albanD commentedon Jul 19, 2023
Not sure what you mean?
Where do you apply this diff?
ezhang887 commentedon Jul 19, 2023
Basically when using "spawn" doing something like
ctx = mp.get_context("spawn"); value = ctx.Value(...)instead ofvalue = mp.Value(...)fixes the segfault in the above example for me.And the same thing for the barrier example in #29855, doing
ctx = mp.get_context("spawn"); barrier = ctx.Barrier(...)instead ofbarrier = mp.Barrier(...)fixes it for me.albanD commentedon Jul 19, 2023
But it is still something we're doing wrong on our end as when torch is not involved, either works fine.
albanD commentedon Jul 19, 2023
btw another way to fix this here is to set
multiprocessing.set_start_method("spawn")before creating your object.I guess the problem is that the user Barrier/Value is created in a fork context (the default) but then used in a spawn context and thus the segfault?
ezhang887 commentedon Jul 19, 2023
Yup also saw that, just wanted to avoid
multiprocessing.set_start_method("spawn")and use the contexts instead since it seemed cleaner.Yup this was what I was thinking.
albanD commentedon Jul 25, 2023
btw this is a know upstream problem in CPython: python/cpython#77377
The simplest pure python repro:
albanD commentedon Jul 25, 2023
@pritamdamania87 so the answer is you should NOT mix different multiprocessing contexts and you should pass the right context to the Dataloader if you want to use a special one!
This is a limitation of CPython that doesn't gracefully handle these mixed context data (see linked issue above).
Is that a satisfiable answer?
pritamdamania87 commentedon Jul 25, 2023
@albanD Thanks a lot for looking into this and yes that is what we are doing currently.
albanD commentedon Jul 26, 2023
Sent a PR to CPython to make it a nicer error: python/cpython#107275
But your code will still have to be updated to be using consistent context.
SuperSecureHuman commentedon Aug 21, 2023
Similar error. I am using PyTorch lightning, the error traces back to a similar style.
Here is a relatable case #4969 (comment)
I am having Python in a different mount than the data. But it's needed for me to do so, because of the amount of data I have.
As far as reproducing this error - it occurs really randomly - Some times in 3rd epoch, sometimes in 8 and so on.
The fact that the same issue is persistent past 5 years is mind twisting
SuperSecureHuman commentedon Aug 21, 2023
Update - Same error without setting num_workers
albanD commentedon Aug 21, 2023
Did you pass in the multiprocessing context as an input to the dataloader as discussed above?
Also if the version without multiprocessing segfaults, that means that the segfault is not the same.
Can you use gdb to get a stack trace for the segfault in the case where num_workers=0?
SuperSecureHuman commentedon Aug 22, 2023
Hey!
No I did not pass multiprocessing context to the dataloader.
I choose to comment here, because - this was the recent issue that was most similar to the one I was getting.
I would need some time to get the stack trace back, since this error is on a remote machine, and the error occurs at random.
I will pipe everything to a file, and attach it here, once I am done with the stack trace
SuperSecureHuman commentedon Aug 22, 2023
Dataloader -
return DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=True, num_workers=0)The "data" part of dataloader is perfectly fine - It trains for a few epochs. So it's mostly not the way how data is being processed.
Update - Ran with num_workers=0, and the error doesn't come up now. As I told, its completely random in my case.
I will run all my runs with gdb, and log the outputs to a file, so that I will share them once I hit the seg fault
SuperSecureHuman commentedon Aug 22, 2023
I have trimmed the output to whereever there was some special output from GDB
Best part is that, only when using multiprocessing, we get to know that error's stack trace (which is from dataloader)
When not using multiple workers, it just says seg fault
/usr/local/src/conda/python-3.10.10/Include/object.h:146--> this is non existent, not sure why its looking here.albanD commentedon Aug 22, 2023
I'm very confused by what you say: "Ran with num_workers=0, and the error doesn't come up now." and "When not using multiple workers, it just says seg fault".
In any case, sounds like this is not this issue. I would suggest you ask on the forum to get help debugging your code: https://discuss.pytorch.org/
SuperSecureHuman commentedon Aug 22, 2023
When I mentioned this -
Ran with num_workers=0that was earlier, when I thought the issue randomly stopped occurring.The 2nd message I sent - where I mentioned
When not using multiple workers, it just says seg faultI actually meant num_workers=0Sorry for the wrong way of telling whatever I want to tell.
In summary - The error still occurs - even when setting workers to 0.
When setting the value of workers more than 1, the PyTorch error trace points back to dataloader.
Again, really sorry if I confused you earlier.
albanD commentedon Aug 22, 2023
This means that the dataloader is not the root cause and your issue doesn't match this one. Please post the details on the forum to discuss this more.
albanD commentedon Aug 23, 2023
This is now fixed in CPython and will nicely raise an error on 3.11+ (once the bugfix patch is released)