Repository navigation
Add PyDict_NewPresized() function #80
Description
Hi,
The private _PyDict_NewPresized(size) function is used by 11 projects of the PyPI top 15,000 projects. The problem is that it miss the unicode_keys parameter which is a hint to announce if keys are Unicode or not. @methane analyzed how _PyDict_NewPresized(size) is used in various projects and suggests adding a public PyDict_NewPresized(size, unicode_keys) function. See python/cpython#139772 for the discussion.
API: PyObject* PyDict_NewPresized(Py_ssize_t size, int unicode_keys)
- Return a new empty dictionary with preallocated items, or
NULLon failure. - size is a hint to preallocate items.
- If unicode_keys is non-zero, optimize lookup for Unicode keys.
Since unicode_keys is an implementation detail, I hesitate to put the function in the PyUnstable category: PyUnstable_Dict_NewPresized(). What do you think?
Vote:
Activity
vstinner commentedon Oct 13, 2025
PyDict_NewPresized()is between 1.03x and 1.10x faster thanPyDict_New()to create a dictionary with Unicode keys.encukou commentedon Oct 13, 2025
11 is not a lot...
For implementation details: why expose unicode_keys, and not for example key-sharing?
Do we guarantee that if you do size insertions, the dict need to reallocate new memory?
How much faster would it be with non-
strkeys?vstinner commentedon Oct 13, 2025
I'm trying to add public functions for the remaining private functions used by Cython and other projects.
But well, it's a micro-optimization, and some people like to gain 5% when possible :-)
PyO3 expressed interest to use this function if it becomes public. Currently, PyO3 tries to avoid private functions.
I have no idea. @methane: Do you have an opinion on that?
I expect that when you call
PyDict_NewPresize(n, unicode_keys), no reallocation is needed to add up to n items. @methane told me that n is just a hint to preallocate, it's not a warranty. But in practice,PyDict_NewPresize()is faster thanPyDict_New()is my benchmark.PyDict_NewPresized() is between 1.04x and 1.10x faster than PyDict_New() to create a dictionary with integer keys. My benchmark only measured the time to create a dictionary.
methane commentedon Oct 14, 2025
This benchmark uses
PyUnicode_FromFormatandPyLong_FromLongin the loop.In real-world use cases like JSON decoding, key or value object creation does happen, but you can expect a bit more benefit in use cases like constructing a dict for
kwargscontains a dozen keys.PyDict_NewPresize()is not suitable for building key-sharing dict.For that, we need to enumerate use cases and design an API that is easy to use for all of them.
Maybe, csv parser, dbapi dictcursor, structured deserializer (msgspec, pydantic, etc...)
encukou commentedon Oct 20, 2025
The discussion is ongoing in the issue/PRs; calling for a vote might have been premature.
vstinner commentedon Nov 5, 2025
The discussion seems to be idle for 2 weeks.
vstinner commentedon Nov 12, 2025
@encukou @erlend-aasland: So what do you think about this API?
encukou commentedon Nov 13, 2025
I cannot support exposing a current CPython implementation detail in the
int unicode_keysargument.I could be convinced to support a
PyObject *key_typeargument instead, if that becomes a performance win for Cython.AFAICS,
@methane's comment still stands.From the 11 projects you mentioned, is there any project other that CPython that would use the new API?
vstinner commentedon Dec 3, 2025
See also #90 which proposes adding
PyDict_FromItems()function.vstinner commentedon Jan 7, 2026
I close this issue in favor of
PyDict_FromItems()API.