The `datetime` C API is not thread-safe

[ZeroIntensity]

Bug report

Bug description:

datetime has a C API that we expose through a global variable. This does not work well under multiple threads. For example, if PyDateTime_IMPORT fails in one thread while another thread is invoking the datetime C API, then that will set PyDateTimeAPI to NULL, leading to NULL pointer dereferences in the original thread.

This global variable also forces us to do some nasty hacks with subinterpreters because it exposes static types. I helped fix this a few months ago by making _datetime static (see this thread), but there are probably other issues that I haven't found yet.

I think a bandaid fix would be to make PyDateTimeAPI a thread-local variable, but I suspect that would break a lot of things, nor would that fix this for subinterpreters. More generally, we could deprecate PyDateTime_IMPORT and PyDateTimeAPI in favor of letting users manage the pointer on their own, but I don't know if we can do this without too many code changes (see #141543 and #83785 for the docs side).

cc @pganssle

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-141563: make PyDateTime_IMPORT thread-safe #144210
• gh-141563: Add missing cast to _PyDateTime_IMPORT() #144667
• gh-141563: Don't test datetime.h with the limited C API #144673
• gh-141563: Fix test_cext on Windows #144677
• gh-141563: Fix test_cppext on macOS #144685
• gh-141563: Enable test_cppext internal C API tests on macOS #144711

[sharktide]

Has anyone found any examples of this on GitHub? I imagine that this might happen because it seems like it would be common to use dateline to measure the time of something executing but at the same time why use datetime itself? There has to be better libs or inbuilt functions for most use cases

[ZeroIntensity]

Yes.

[kumaraditya303]

I think we can make it thread safe with atomics, basically create a exchange loop and only initialize it if it is not initialized already.

Something like this should work:

diff --git a/Include/datetime.h b/Include/datetime.h
index ed36e6e48c8..17c85b19e11 100644
--- a/Include/datetime.h
+++ b/Include/datetime.h
@@ -197,7 +197,15 @@ typedef struct {
 static PyDateTime_CAPI *PyDateTimeAPI = NULL;
 
 #define PyDateTime_IMPORT \
-    PyDateTimeAPI = (PyDateTime_CAPI *)PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0)
+    do
+    {
+        PyDateTime_CAPI *capi = (PyDateTime_CAPI *)PyCapsule_Import(PyDateTime_CAPSULE_NAME, 0); \
+        if (capi && _Py_atomic_load_ptr(&PyDateTimeAPI) == NULL) { \
+            PyDateTime_CAPI *expected_api = NULL; \
+            _Py_atomic_compare_exchange_ptr(&PyDateTimeAPI, &expected_api, capi); \
+        }
+    } while (0)
+
 
 /* Macro for access to the UTC singleton */
 #define PyDateTime_TimeZone_UTC PyDateTimeAPI->TimeZone_UTC