Skip to content

gh-90815: Add mimalloc memory allocator #109914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 24 commits into from
Oct 30, 2023
Merged

gh-90815: Add mimalloc memory allocator #109914

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
79b2cf4
Add mimalloc v2.0.9
colesbury Feb 10, 2020
0288841
Upgrade to mimalloc 2.12
DinoV Sep 21, 2023
612c35e
Fix mimalloc formatting
DinoV Sep 27, 2023
359f90a
Don't export symbol
DinoV Sep 26, 2023
7117b06
Blurb
DinoV Sep 26, 2023
6bd9b0e
mimalloc: enable as allocator option
DinoV Sep 25, 2023
2165c5f
Add includes to analyzer
DinoV Sep 27, 2023
b6bddbf
mimalloc: minimal changes for use in Python
colesbury Feb 11, 2020
7f7f4fe
Add mimalloc MIT license
DinoV Sep 29, 2023
7368010
Log mimalloc in Lib/test/pythoninfo.py
DinoV Oct 2, 2023
8975bd1
Use macro defs for exports as done in https://github.com/python/cpyth…
DinoV Oct 2, 2023
8a71892
Fix formatting
DinoV Oct 2, 2023
d725922
Fix ./python -m test test_sys -R 3:3
DinoV Oct 6, 2023
2cb4aaf
Move mimalloc headers to internal, remove re-defs, update smelly
DinoV Oct 10, 2023
82305dc
Fix header locations on Windows
DinoV Oct 11, 2023
2d0158a
Try again to fix windows headers
DinoV Oct 12, 2023
708cc66
Make memory sanitizer happy
DinoV Oct 16, 2023
526a08d
Keep pymalloc the default
DinoV Oct 23, 2023
3d43b3a
Update Lib/test/pythoninfo.py
vstinner Oct 30, 2023
3261bcb
license.rst: remove empty line at the end
vstinner Oct 30, 2023
99c4a24
Reduce Makefile.pre.in indentation
vstinner Oct 30, 2023
e2e36dc
obmalloc.c: indent
vstinner Oct 30, 2023
62b7b83
Split _PyObject_DebugMallocStats() in sub-functions
vstinner Oct 30, 2023
ab829b3
Fix memory doc
vstinner Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move mimalloc headers to internal, remove re-defs, update smelly 
to allow mi malloc symbols in static lib, but not shared lib

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
  • Loading branch information
@DinoV
DinoV committed Oct 12, 2023
commit 2cb4aaf1d069dd07d527827a95d845f3825e852b
0 Include/mimalloc/mimalloc.h → Include/internal/mimalloc/mimalloc.h
View file
Open in desktop
File renamed without changes.
0 Include/mimalloc/mimalloc/atomic.h → Include/internal/mimalloc/mimalloc/atomic.h
View file
Open in desktop
File renamed without changes.
0 Include/mimalloc/mimalloc/internal.h → ...ude/internal/mimalloc/mimalloc/internal.h
View file
Open in desktop
File renamed without changes.
0 Include/mimalloc/mimalloc/prim.h → Include/internal/mimalloc/mimalloc/prim.h
View file
Open in desktop
File renamed without changes.
0 Include/mimalloc/mimalloc/track.h → Include/internal/mimalloc/mimalloc/track.h
View file
Open in desktop
File renamed without changes.
0 Include/mimalloc/mimalloc/types.h → Include/internal/mimalloc/mimalloc/types.h
View file
Open in desktop
File renamed without changes.
282 changes: 0 additions & 282 deletions Include/internal/pycore_mimalloc.h
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 23 additions & 7 deletions Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ CONFIGURE_LDFLAGS= @LDFLAGS@
# command line to append to these values without stomping the pre-set
# values.
PY_CFLAGS= $(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)
PY_CFLAGS_NODIST=$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal -I$(srcdir)/Include/mimalloc
PY_CFLAGS_NODIST=$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal -I$(srcdir)/Include/internal/mimalloc
# Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py to
# be able to build extension modules using the directories specified in the
# environment variables
Expand Down Expand Up @@ -342,12 +342,12 @@ IO_OBJS= \

MIMALLOC_HEADERS= \
$(srcdir)/Include/internal/pycore_mimalloc.h \
$(srcdir)/Include/mimalloc/mimalloc.h \
$(srcdir)/Include/mimalloc/mimalloc/atomic.h \
$(srcdir)/Include/mimalloc/mimalloc/internal.h \
$(srcdir)/Include/mimalloc/mimalloc/prim.h \
$(srcdir)/Include/mimalloc/mimalloc/track.h \
$(srcdir)/Include/mimalloc/mimalloc/types.h
$(srcdir)/Include/internal/mimalloc/mimalloc.h \
$(srcdir)/Include/internal/mimalloc/mimalloc/atomic.h \
$(srcdir)/Include/internal/mimalloc/mimalloc/internal.h \
$(srcdir)/Include/internal/mimalloc/mimalloc/prim.h \
$(srcdir)/Include/internal/mimalloc/mimalloc/track.h \
$(srcdir)/Include/internal/mimalloc/mimalloc/types.h


##########################################################################
Expand Down Expand Up @@ -1552,6 +1552,22 @@ Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c $(UNICODE_DEPS)
Objects/dictobject.o: $(srcdir)/Objects/stringlib/eq.h
Objects/setobject.o: $(srcdir)/Objects/stringlib/eq.h

Objects/obmalloc.o: $(srcdir)/Objects/mimalloc/alloc.c \
$(srcdir)/Objects/mimalloc/alloc-aligned.c \
$(srcdir)/Objects/mimalloc/alloc-posix.c \
$(srcdir)/Objects/mimalloc/arena.c \
$(srcdir)/Objects/mimalloc/bitmap.c \
$(srcdir)/Objects/mimalloc/heap.c \
$(srcdir)/Objects/mimalloc/init.c \
$(srcdir)/Objects/mimalloc/options.c \
$(srcdir)/Objects/mimalloc/os.c \
$(srcdir)/Objects/mimalloc/page.c \
$(srcdir)/Objects/mimalloc/random.c \
$(srcdir)/Objects/mimalloc/segment.c \
$(srcdir)/Objects/mimalloc/segment-map.c \
$(srcdir)/Objects/mimalloc/stats.c \
$(srcdir)/Objects/mimalloc/prim/prim.c

Objects/mimalloc/page.o: $(srcdir)/Objects/mimalloc/page-queue.c

.PHONY: regen-cases
Expand Down
13 changes: 10 additions & 3 deletions Tools/build/smelly.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@
if sys.platform == 'darwin':
ALLOWED_PREFIXES += ('__Py',)

# mimalloc doesn't use static, but it's symbols are not exported
# from the shared library. They do show up in the static library
# before its linked into an executable.
ALLOWED_STATIC_PREFIXES = ('mi_', '_mi_')

@vstinner vstinner Oct 21, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for distinguishing local and exported symbols. It's good that "mi" and "mi" prefixes are only allowed for local symbols.


# "Legacy": some old symbols are prefixed by "PY_".
EXCEPTIONS = frozenset({
'PY_TIMEOUT_MAX',
Expand Down Expand Up @@ -59,7 +64,7 @@ def get_exported_symbols(library, dynamic=False):
return stdout


def get_smelly_symbols(stdout):
def get_smelly_symbols(stdout, dynamic=False):
smelly_symbols = []
python_symbols = []
local_symbols = []
Expand All @@ -77,7 +82,9 @@ def get_smelly_symbols(stdout):
symbol = parts[-1]
result = '%s (type: %s)' % (symbol, symtype)

if symbol.startswith(ALLOWED_PREFIXES) or symbol in EXCEPTIONS:
if (symbol.startswith(ALLOWED_PREFIXES) or
symbol in EXCEPTIONS or
(not dynamic and symbol.startswith(ALLOWED_STATIC_PREFIXES))):
python_symbols.append(result)
continue

Expand All @@ -95,7 +102,7 @@ def get_smelly_symbols(stdout):

def check_library(library, dynamic=False):
nm_output = get_exported_symbols(library, dynamic)
smelly_symbols, python_symbols = get_smelly_symbols(nm_output)
smelly_symbols, python_symbols = get_smelly_symbols(nm_output, dynamic)

if not smelly_symbols:
print(f"OK: no smelly symbol found ({len(python_symbols)} Python symbols)")
Expand Down