Make weakrefs to immutable objects thread-safe#75
Make weakrefs to immutable objects thread-safe#75kulisak12 wants to merge 6 commits intofxpl:immutable-mainfrom
Conversation
Objects/weakrefobject.c
Outdated
| if (_Py_IsImmutable(obj)) { | ||
| // Weakrefs to immutable objects can be shared across interpreters. | ||
| // They need to be marked as immutable to turn on atomic refcounts. | ||
| _PyImmutability_Freeze(_PyObject_CAST(newref)); |
There was a problem hiding this comment.
The newref here can refer to a weakref with a callback. Freezing a callback would change the semantics of that function. Is this needed? I think we only want to freeze it, if the callback is none because that's also the only case where the ref might be shared. If I understand this correctly?
Objects/weakrefobject.c
Outdated
| if (_Py_IsImmutable(obj)) { | ||
| // Weakrefs to immutable objects can be shared across interpreters. | ||
| // They need to be marked as immutable to turn on atomic refcounts. | ||
| _PyImmutability_Freeze(_PyObject_CAST(newref)); | ||
| } |
There was a problem hiding this comment.
Freezing a weak ref makes it a strong reference in the freeze code!
There was a problem hiding this comment.
Right, I forgot about that. If a weak ref is part of an SCC this makes a lot of sense, but I'm considering if we want to keep this behaviour for normal references. I fully agree that the object they're referencing should also be frozen. Just not sure about the strong ref part.
ec30f37 to
5261949
Compare
5261949 to
c3b5604
Compare
|
I consider this ready for code review now. The big comment in the beginning of I haven't been able to test it over multiple interpreters (my attempts crash even on One unsolved issue is that weakrefs with callbacks have their refcount pre-emptively incremented, which prevents them from dying if they are a part of an SCC. We likely no longer need |
xFrednet
left a comment
There was a problem hiding this comment.
This is amazing! I have one tiny nit, but then I think it would be good to go.
| for _ in range(10): | ||
| if self._called: | ||
| return True |
There was a problem hiding this comment.
This seems like an unstable test waiting to happen. I think it would be better to have a small delay here
| for _ in range(10): | |
| if self._called: | |
| return True | |
| import time | |
| for _ in range(10): | |
| if self._called: | |
| return True | |
| time.sleep(0.01) |
I also assume that time.sleep releases the GIL making it even more likely to handle the callback
| PyThreadState* tstate_old = PyThreadState_Swap(tstate_target); | ||
| int schedule_res = Py_AddPendingCall(weakref_call_callbacks, (void*)pending); | ||
| PyThreadState_Swap(tstate_old); | ||
| if (schedule_res != 0) { |
There was a problem hiding this comment.
This looks so cursed, but I'm super happy that you found this and think that it's actually super clean given the task at hand! :D
Currently untested, I want to write some tests tomorrow.
The following is implemented:
Deallocation of the SCC is not yet handled.