In many cases when we want to convert some data into an existing type with msgspec.convert(data, typ), typ is already known in the import time.
At the same time, not everything can be a Struct. For example: dict[str, list[int]]. Or in django-modern-rest, we allow all possible return types / request bodies, which are user-defined but known in the import time.
So, this part becomes the slowest part in the parsing:
|
/* Avoid allocating a new TypeNode for struct types */ |
|
if (ms_is_struct_cls(pytype)) { |
|
PyObject *info = StructInfo_Convert(pytype); |
|
if (info == NULL) return NULL; |
|
bool array_like = ((StructMetaObject *)pytype)->array_like == OPT_TRUE; |
|
TypeNodeSimple type; |
|
type.types = array_like ? MS_TYPE_STRUCT_ARRAY : MS_TYPE_STRUCT; |
|
type.details[0].pointer = info; |
|
PyObject *out = convert(&state, obj, (TypeNode *)(&type), NULL); |
|
Py_DECREF(info); |
|
return out; |
|
} |
|
|
|
TypeNode *type = TypeNode_Convert(pytype); |
|
if (type == NULL) return NULL; |
|
PyObject *out = convert(&state, obj, type, NULL); |
|
TypeNode_Free(type); |
|
return out; |
|
} |
What if we can prepare TypeInfo in advance, like TypeAdapter does in pydantic https://pydantic.dev/docs/validation/dev/concepts/type_adapter
The API can look similar to existing Decoder objects:
|
typedef struct Decoder { |
|
PyObject_HEAD |
|
PyObject *orig_type; |
|
|
|
/* Configuration */ |
|
TypeNode *type; |
|
char strict; |
|
PyObject *dec_hook; |
|
PyObject *ext_hook; |
|
} Decoder; |
Which already contain the needed TypeInfo. So, it would look like so:
from typing import Any
import msgspec
converter = msgpec.Converter(dict[str, list[int]], strict=True, dec_hook=None)
def user_callback(raw_data: Any) -> dict[str, list[int]]:
# called per user request, hot path
return converter.convert(raw_data)It will only require:
- Adding a new class
- Prepare
TypeInfo in advance
- Reuse existing
msgspec_convert logic
- Documenting and testing this new feature
I will send a PR soon :)
In many cases when we want to convert some data into an existing type with
msgspec.convert(data, typ),typis already known in the import time.At the same time, not everything can be a
Struct. For example:dict[str, list[int]]. Or indjango-modern-rest, we allow all possible return types / request bodies, which are user-defined but known in the import time.So, this part becomes the slowest part in the parsing:
msgspec/src/msgspec/_core.c
Lines 22578 to 22596 in 2e2c863
What if we can prepare
TypeInfoin advance, likeTypeAdapterdoes in pydantic https://pydantic.dev/docs/validation/dev/concepts/type_adapterThe API can look similar to existing
Decoderobjects:msgspec/src/msgspec/_core.c
Lines 15068 to 15077 in 2e2c863
Which already contain the needed
TypeInfo. So, it would look like so:It will only require:
TypeInfoin advancemsgspec_convertlogicI will send a PR soon :)