ENH: Add madvise for memmap objects #13172
Description
Seems like a problem with large mmap memory page size
Should we add an interface like madvise to tell system how the memmapped array would be used?
This problem doesn't exist on Windows 10.
Reproducing code example:
Version 1(~100x slower on Linux, fast enough on Windows):
import numpy as np
import time
large_data = np.memmap("/home/user/100GB.mmap", dtype=np.float32, mode="r", order="C")
entry_count = large_data.size // 256
large_data = large_data.reshape(entry_count, 256)
t = time.time()
large_data[np.random.randint(0, entry_count, size=32768)].copy()
print(time.time() - t)Version 2(fast enough on Linux):
import numpy as np
import time
import ctypes
large_data = np.memmap("/home/user/100GB.mmap", dtype=np.float32, mode="r", order="C")
madvise = ctypes.CDLL("libc.so.6").madvise
madvise.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]
madvise.restype = ctypes.c_int
assert madvise(large_data.ctypes.data, large_data.size * large_data.dtype.itemsize, 1) == 0, "MADVISE FAILED" # 1 means MADV_RANDOM
entry_count = large_data.size // 256
large_data = large_data.reshape(entry_count, 256)
t = time.time()
large_data[np.random.randint(0, entry_count, size=32768)].copy()
print(time.time() - t)Numpy/Python version information:
1.16.2 3.7.2 (default, Dec 30 2018, 16:18:15) [GCC]
System information:
Linux: Linux 5.0.2 x86_64 + Samsung 860 EVO SSD with BtrFS
Windows: Windows 10 1809 + Samsung 860 EVO SSD with NTFS
Activity
seberg commentedon Mar 22, 2019
Numpy's memmap is a pretty shallow wrapper around pythons mmap. Which is I guess why Antoine Pitrou (and others) seem to be looking at adding madvise support to the python mmap object:
python/cpython#6172
The easiest would be probably for us to add it to numpy as soon as python supports it, which also means we have to worry less about the different OSs. As a side note, we already use madvise for (large?) allocations. If it is not very ugly, maybe adding it to numpy separately could be an option as well, though.
mhvk commentedon Mar 22, 2019
I asked over there about the status; it seems abandoned, but very easy to move forward.
a-deeb commentedon Mar 25, 2019
Hi, I would like to help out, I just made a pull request and added the file but I'm not sure if its in the correct directory
hvasbath commentedon Aug 24, 2019
Hi, apparently it got merged into cython now! What are the plans to continue integration into the numpy memmap? It would be great to have!
eric-wieser commentedon Aug 24, 2019
Thanks for pinging this issue!
The easiest thing to do would be to write a wrapper method just like we did for
flush:numpy/numpy/core/memmap.py
Lines 297 to 313 in 77b421d
You'd probably want to make it conditional on the python version.
oborchers commentedon Oct 28, 2019
Is there any progress wrt to the issue? Currently facing the need to randomly index about 2tb of memmaps. The initial solution by @tuxzz worked as a bugfix so far. Many thanks
mattip commentedon Oct 28, 2019
A PR based on the comment above would be welcome since this has been merged to python3.8. If done quickly we might be able to get it into numpy 1.18. Don't forget to add tests.
hvasbath commentedon Dec 19, 2019
any news on that? I would be very interested in using it, but I fear I am lacking the knowledge to properly implement it myself...
seberg commentedon Dec 19, 2019
@hvasbath you can likely take a shot at it. If you check https://github.com/python/cpython/pull/6172/files you will see that on python 3.8 there is a simple method which can just be forwarded.
[-]Random access a huge numpy.memmap array is incredible slow on Linux.[/-][+]ENH: Add `madvise` for `memmap` objects[/+]ENH: Add madvise for memmap objects
ENH: Add madvise for memmap objects
Laubeee commentedon Mar 7, 2025
I'd appreciate this feature... I suppose in the meantime I can just use
arr._mmap.madivse()