Skip to content

Make weakrefs to immutable objects thread-safe#75

Open
kulisak12 wants to merge 6 commits intofxpl:immutable-mainfrom
kulisak12:immutable-weakrefs
Open

Make weakrefs to immutable objects thread-safe#75
kulisak12 wants to merge 6 commits intofxpl:immutable-mainfrom
kulisak12:immutable-weakrefs

Conversation

@kulisak12
Copy link

Currently untested, I want to write some tests tomorrow.

The following is implemented:

  • using a global lock to synchronize operations on the weakref list
  • forwarding refcount checks to the SCC root
  • the operations on the shared weakref objects are thread-safe

Deallocation of the SCC is not yet handled.

if (_Py_IsImmutable(obj)) {
// Weakrefs to immutable objects can be shared across interpreters.
// They need to be marked as immutable to turn on atomic refcounts.
_PyImmutability_Freeze(_PyObject_CAST(newref));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newref here can refer to a weakref with a callback. Freezing a callback would change the semantics of that function. Is this needed? I think we only want to freeze it, if the callback is none because that's also the only case where the ref might be shared. If I understand this correctly?

Comment on lines +475 to +479
if (_Py_IsImmutable(obj)) {
// Weakrefs to immutable objects can be shared across interpreters.
// They need to be marked as immutable to turn on atomic refcounts.
_PyImmutability_Freeze(_PyObject_CAST(newref));
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Freezing a weak ref makes it a strong reference in the freeze code!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I forgot about that. If a weak ref is part of an SCC this makes a lot of sense, but I'm considering if we want to keep this behaviour for normal references. I fully agree that the object they're referencing should also be frozen. Just not sure about the strong ref part.

@kulisak12 kulisak12 marked this pull request as draft March 6, 2026 08:56
@kulisak12 kulisak12 force-pushed the immutable-weakrefs branch 2 times, most recently from ec30f37 to 5261949 Compare March 12, 2026 12:39
@kulisak12 kulisak12 force-pushed the immutable-weakrefs branch from 5261949 to c3b5604 Compare March 12, 2026 13:20
@kulisak12 kulisak12 marked this pull request as ready for review March 12, 2026 13:56
@kulisak12
Copy link
Author

kulisak12 commented Mar 12, 2026

I consider this ready for code review now.
I believe there is one more potential bug, but I can work on that one in the meantime.

The big comment in the beginning of weakrefobject.c summarizes the changes made.

I haven't been able to test it over multiple interpreters (my attempts crash even on immutable-main).
In particular, I have no idea if the code to schedule a callback on a different interpreter is correct.
However, it worked on a single interpreter, even before making the last commit, which optimizes this case.

One unsolved issue is that weakrefs with callbacks have their refcount pre-emptively incremented, which prevents them from dying if they are a part of an SCC.
It could theoretically be solved with fine-grained conditionals over whether the weakref is frozen or not.
Fred also mentioned he has another solution for making weakrefs with callbacks thread-safe once IP-local objects are implemented.

We likely no longer need scc_details.has_weakreferences, but I decided to keep it for now.

Copy link
Collaborator

@xFrednet xFrednet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing! I have one tiny nit, but then I think it would be good to go.

Comment on lines +27 to +29
for _ in range(10):
if self._called:
return True
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like an unstable test waiting to happen. I think it would be better to have a small delay here

Suggested change
for _ in range(10):
if self._called:
return True
import time
for _ in range(10):
if self._called:
return True
time.sleep(0.01)

I also assume that time.sleep releases the GIL making it even more likely to handle the callback

Comment on lines +925 to +928
PyThreadState* tstate_old = PyThreadState_Swap(tstate_target);
int schedule_res = Py_AddPendingCall(weakref_call_callbacks, (void*)pending);
PyThreadState_Swap(tstate_old);
if (schedule_res != 0) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks so cursed, but I'm super happy that you found this and think that it's actually super clean given the task at hand! :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants