Repository navigation
Fixed #29478 -- Added support for mangled names to cached_property. - #10033
Conversation
4b6aca6 to
bf0148d
Compare
2141388 to
e7b457b
Compare
There was a problem hiding this comment.
I think this should throw a warning on < PY36 where re.startswith('__') and not re.endswith('__')
There was a problem hiding this comment.
Not sure about the best way to do that though
There was a problem hiding this comment.
I think a plain warning would be fine.
There was a problem hiding this comment.
People like to copy this file into their non Django related files. I'd like to avoid importing Django here for that reason
There was a problem hiding this comment.
Could be a plain warnings.warn
Also not sure that the 'people like to copy this file' argument holds for Django's codebase. There's pip install cached-property for an implementation of this outside of Django.
There was a problem hiding this comment.
actually this is redundant because PY35 is no longer supported:
eg I can't do: if django.utils.version.PY35 and name is None and func.__name__.startwith('__') and not func.__name__.startwith('__') anyway
adamchainz
left a comment
There was a problem hiding this comment.
This will also need a release note in the 2.2 release note docs, and maybe we need to mark name for deprecation starting with the future version of Django that supports only 3.6+ (which might be 3.0?)
There was a problem hiding this comment.
Wrap this into previous paragraph so we only have one explanation of name?
On Python <3.6, the ``name`` argument should be passed to make a cached property from another method without a name collision, e.g. ``url = cached_property(get_absolute_url, name='url')``.
There was a problem hiding this comment.
I think a plain warning would be fine.
There was a problem hiding this comment.
I think we'd normally call this check_cached_property or similar
Also I'd prefer a longer class name than A whilst we're refactoring these tests
There was a problem hiding this comment.
We typically follow the camel casing of unittest assertions, e.g. assertCachedPropertyWorks. Try to split out test refactoring in a separate PR to ease review.
There was a problem hiding this comment.
currently the refactorings are in separate commits, here's the copy/paste version: 0870388
There was a problem hiding this comment.
@timgraham I've also included a call to with self.subTest
e7b457b to
13f1b2e
Compare
13f1b2e to
260266f
Compare
There was a problem hiding this comment.
Can you just do name.startswith('__') and not name.endswith('__')? Simpler is better
c02fa92 to
96ac758
Compare
|
release note added |
f4c6661 to
7c03011
Compare
timgraham
left a comment
There was a problem hiding this comment.
Should it be a warning or an error? In either case, please test the warning/error case.
There was a problem hiding this comment.
I'm not sure what Adam was thinking of with regards to the release note, but we don't usually document changes in django.utils to be "minor features".
There was a problem hiding this comment.
Do you know where it should go?
There was a problem hiding this comment.
Most likely it doesn't need a release unless there's some backwards incompatible change.
There was a problem hiding this comment.
I've seen a lot of use of @cached_property so I thought it would count, but yeah let's not worry about it.
There was a problem hiding this comment.
It should be documented when 3.5 is dropped and name is deprecated
Probably should not be an Error, because just upgrading to Python 3.6 will fix it |
7c03011 to
c22b6f7
Compare
|
Can't we simply construct mangled name in |
|
@sir-sigurd nope, because you don't know which class to mangle the name with |
There was a problem hiding this comment.
yeah but it's the wrong cls
There was a problem hiding this comment.
I think it's the right class:
In [38]: class desc:
...: def __get__(self, instance, cls):
...: if instance is None:
...: return self
...: return instance.__dict__['_%s__mangled' % cls.__name__]
In [39]: class A:
...: d = desc()
...:
...: def __init__(self):
...: self.__mangled = 42
...:
In [40]: A().d
Out[40]: 42
There was a problem hiding this comment.
Try with a subclass of A
There was a problem hiding this comment.
You might want to add tests for the subclass case @graingert to make it clear.
There was a problem hiding this comment.
@graingert Probably it doesn't worth it, but still doable :-)
In [85]: class desc:
...: def __init__(self, f):
...: self.name = f.__name__
...:
...: def __get__(self, instance, cls):
...: if instance is None:
...: return self
...:
...: for cls in cls.__mro__:
...: name = '_%s%s' % (cls.__name__, self.name)
...: if cls.__dict__.get(name) is self:
...: return name
...:
...: class A:
...: @desc
...: def __x(self):
...: pass
...:
...:
...: class B(A):
...: pass
...:
In [86]: a = A()
...: a._A__x
...:
Out[86]: '_A__x'
In [87]: b = B()
...: b._A__x
...:
Out[87]: '_A__x'
1bb4639 to
59f1483
Compare
78dd2a9 to
d0e1111
Compare
There was a problem hiding this comment.
You'll want to branch off greater or equal to Python 3.6 here.
There was a problem hiding this comment.
that's how this works already
There was a problem hiding this comment.
You'll want to branch of lesser than Python 3.6 here.
There was a problem hiding this comment.
that's how this works already
There was a problem hiding this comment.
You might want to add tests for the subclass case @graingert to make it clear.
There was a problem hiding this comment.
I feel like this helper is making the assertion harder to reason about. Inlining getattr(source, attr) calls would be easier to read IMO.
There was a problem hiding this comment.
it gets very long winded when you add getattr(source, attr) eveywhere
There was a problem hiding this comment.
What's the rationale for changing this test class name? It looks like unnecessary noise to me.
There was a problem hiding this comment.
@adamchainz said to do it: #10033 (comment)
There was a problem hiding this comment.
Ditto about the class __name__ change.
There was a problem hiding this comment.
You'll want to branch off < 3.6.
There was a problem hiding this comment.
that's how this works
043a48c to
6e5402a
Compare
|
If I understand correctly, the documentation for cached_property, including the warning added in 0919973 should be updated to say that |
|
Can you suggest a wording? |
There was a problem hiding this comment.
Might make sense to check explicitly set name too, because '__name' obviously will not work.
There was a problem hiding this comment.
class A:
__print = cached_property(print, '__print')
This will not work and we can easily detect it too.
There was a problem hiding this comment.
What should the condition be?
There was a problem hiding this comment.
@sir-sigurd can you advise about the changes to address your comments?
There was a problem hiding this comment.
I'll push my edits tomorrow.
There was a problem hiding this comment.
Why the warning, but not an exception?
There was a problem hiding this comment.
I also still don't understand why it's useful to allow writing code that doesn't work.
There was a problem hiding this comment.
well it does work, it just behaves as though @cached_property were not there. And it writes to some self.__ attr
There was a problem hiding this comment.
I think silently failing to cache the property should be considered not working.
There was a problem hiding this comment.
I would consider that as not working
There was a problem hiding this comment.
What should the exception be?
There was a problem hiding this comment.
ValueError maybe since passing along name can fix it?
There was a problem hiding this comment.
Might make sense to raise exception in this case:
class Test:
@cached_property
def a(self):
pass
b = a
6e5402a to
89e7707
Compare
|
I added more checks, |
384a904 to
91b16f7
Compare
|
Looks like Jenkins didn't run correctly on the latest commits. After I squashed and pushed some cosmetic changes, there are errors like "TypeError: Cannot assign the same cached_property to two different names ('is_postgresql_9_5' and 'has_select_for_update_skip_locked')" (which were there before my latest changes). |
|
We need to fix it because some attributes are not cached. For example here: django/django/db/backends/postgresql/features.py Lines 56 to 74 in 8aad4a3 only Probably we need to add it to release notes as breaking change and give example when it doesn't work as expected for Python 3.5 users. |
|
Either in this PR or in another one seems fine. Thanks! |
There was a problem hiding this comment.
FWIW the TypeError raised here would be backward incompatible. Such a pattern is even used in Django core.
django/django/db/backends/postgresql/features.py
Lines 56 to 74 in 0ce2ad9
There was a problem hiding this comment.
Yea, as Sergey said above, there's a bug where such attributes aren't cached.
There was a problem hiding this comment.
Ahh right just read through the whole discussion.
I wonder if this could be an opportunity to solve the issue.
e.g.
def __set_name__(self, owner, name):
if self.name is None:
self.name = name
self.func = self.real_func
elif name != self.name:
# Re-assignment of a cached property to a class
# creates a new instance.
instance = type(self)(self.real_func, name)
setattr(owner, name, instance)That would make
@cached_property
def is_postgresql_9_5(self):
return self.connection.pg_version >= 90500
has_select_for_update_skip_locked = is_postgresql_9_5The equivalent of
@cached_property
def is_postgresql_9_5(self):
return self.connection.pg_version >= 90500
has_select_for_update_skip_locked = cached_property(
is_postgresql_9_5.real_func, 'has_select_for_update_skip_locked'
)And address the bug preventing the __dict__ level caching bug discovered by @sir-sigurd.
There was a problem hiding this comment.
I wonder if this could be an opportunity to solve the issue.
I considered it, but:
- it doesn't fix caching on Python 3.5, raising
TypeErroron Python 3.6 could useful when code is tested against both versions (for example we wouldn't know that we use such pattern that works incorrectly on Python 3.5) - I'm not sure if we want to make it work this way, we might want this instead:
@cached_property
def is_postgresql_9_5(self):
return self.connection.pg_version >= 90500
has_select_for_update_skip_locked = property(
attrgetter('is_postgresql_9_5')
)This way touching both is_postgresql_9_5 and has_select_for_update_skip_locked will cause is_postgresql_9_5.real_func be called just once.
We might add cached_property.alias() method for functionality.
|
I'll add a release note to the first commit to describe the change of not allowing cached property aliases (which weren't cached anyway) and recommended the |
I'm going to implement |
|
@sir-sigurd one thing to consider with Not a blocker here as it will be implemented in another PR but definitely something to consider documenting or adapting to support both. Something also worth nothing is that the suggested implementation will probably be a bit slower to retrieve a cached value from an alias because |
cb931ab to
8810cfa
Compare
There was a problem hiding this comment.
Looks like this will raise even if name kwarg is set
There was a problem hiding this comment.
Did you mean to put the name kwarg on the decorator?
There was a problem hiding this comment.
I think it's intended. Based on the documentation, a valid name has to be in the form _Class__attribute. That said, unless I've made a mistake it doesn't seem to work (Python 3.5):
from django.utils.functional import cached_property
class Obj:
@cached_property
def __value1(self, name='_Obj__value1'):
pass(gives ValueError: cached_property does not work with mangled methods on Python < 3.6 without the appropriate name argument.)
Is it a bug or did I misunderstand something, Sergey?
There was a problem hiding this comment.
You're putting it on the decorated method, rather than as a decorator kwarg
There was a problem hiding this comment.
args on property methods are redundant
There was a problem hiding this comment.
Got it. It seems there's still a bug since the test isn't correct. For example, the documented example doesn't require the name argument despite the fact that it says it does:
class Person:
def get_friends(self):
pass
__friends = cached_property(get_friends)
No exception for Python 3.5 but docs say name='_Person__friends' is required.
There was a problem hiding this comment.
No exception for Python 3.5 but docs say name='_Person__friends' is required.
It's required because it won't be cached and no exception is raised because there is no way to detect it.
1400048 to
89c9aff
Compare
Co-Authored-By: Sergey Fedoseev <fedoseev.sergey@gmail.com>
89c9aff to
0607699
Compare
https://code.djangoproject.com/ticket/29478