Make _Py_ThreadId() work on PowerPC, IBM Z, etc.

@vstinner

Feature or enhancement

Currently, _Py_ThreadId() is only defined in Py_GIL_DISABLED builds and does not compile on PowerPC or IBM Z. It would be convenient to make the functionality available in general: it will be useful to implement thread-safe recursive locks. (Currently our recursive lock implementations rely on the GIL for thread-safety.)

There are two parts to this:

We should use __builtin_thread_pointer() when available
We should fall back to a function that returns the address of a thread-local variable if no other fast implementation is available (instead of erroring out).

The fallback (2) will be a bit slower, but it provides a portable implementation.

Note that checking if __builtin_thread_pointer() is available is not trivial. __has_builtin(__builtin_thread_pointer) is not useful (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96952). You need some combination of platform and GCC/clang version check.

cc @vstinner @corona10

Linked PRs

gh-112535: Add test on _Py_ThreadId() (#112709)

Merge remote-tracking branch 'upstream/main' into pythongh-112535-pow…

gh-112535: Update _Py_ThreadId() to support PowerPC (gh-112624)

pythongh-112535: Add comment for ppc32/64 registers

gh-112535: Add comment for ppc32/64 registers (gh-112746)

pythongh-112535: Update _Py_ThreadId() to support s390/s390x

gh-112535: Update _Py_ThreadId() to support s390/s390x (gh-112751)

@colesbury

@colesbury Do we need to support more platforms or we can close the issue?

I think we still want the cross-platform fallback that returns the address of a thread-local variable. That will probably need to be implemented in a .c file because accessing thread-local variables across libraries can cause issues on some platforms.

FYI, PEP 11 defines which platform we should cover :)
According to PEP 11, WASM is left but IIRC, Support thread feature from WASM side is quite early stage: https://github.com/WebAssembly/wasi-threads
https://peps.python.org/pep-0011/

Adding a fallback will ease the maintenance burden and also work with WASM. In general, I think we want portable implementations when possible even if the platforms listed in PEP 11 ought to work without them.

We can't use _Py_ThreadId() outside of the --disable-gil builds right now. That makes it harder to implement an efficient recursive lock. We still will not want to expose _Py_ThreadId() outside of --disable-gil builds, but it will be convenient to use it in .c files like Python/lock.c.

cpython/Python/thread_pthread.h

Lines 380 to 406 in 7595d47

    
           #ifdef __APPLE__ 
        
               uint64_t native_id; 
        
               (void) pthread_threadid_np(NULL, &native_id); 
        
           #elif defined(__linux__) 
        
               pid_t native_id; 
        
               native_id = syscall(SYS_gettid); 
        
           #elif defined(__FreeBSD__) 
        
               int native_id; 
        
               native_id = pthread_getthreadid_np(); 
        
           #elif defined(__FreeBSD_kernel__) 
        
               long native_id; 
        
               syscall(SYS_thr_self, &native_id); 
        
           #elif defined(__OpenBSD__) 
        
               pid_t native_id; 
        
               native_id = getthrid(); 
        
           #elif defined(_AIX) 
        
               tid_t native_id; 
        
               native_id = thread_self(); 
        
           #elif defined(__NetBSD__) 
        
               lwpid_t native_id; 
        
               native_id = _lwp_self(); 
        
           #elif defined(__DragonFly__) 
        
               lwpid_t native_id; 
        
               native_id = lwp_gettid(); 
        
           #endif 
        
               return (unsigned long) native_id; 
        
           }

Well, we already implemented PyThread_get_thread_native_id for getting native thread id. Do you want to use same approach as the fallback?
If yes, Which platforms do we have to care about? (Linux Only? or other things too?)

@corona10

@corona10, no I don't think we should use PyThread_get_thread_native_id. I think we should do what I wrote as part (2) in the issue: "return the address of a thread-local variable"

Got it, I will send a PR soon. I read TLS spec in here and now I understand what you want to say.(maybe?)

@corona10

@corona10, here is an example of what I mean in mimalloc, which uses the same strategy on platforms where it doesn't implement a faster _mi_prim_thread_id():

cpython/Include/internal/mimalloc/mimalloc/prim.h

Line 129 in 9cdf05b

    
           extern mi_decl_thread mi_heap_t* _mi_heap_default;  // default heap to allocate from

cpython/Include/internal/mimalloc/mimalloc/prim.h

Lines 224 to 226 in 9cdf05b

    
           static inline mi_threadid_t _mi_prim_thread_id(void) mi_attr_noexcept { 
        
             return (uintptr_t)&_mi_heap_default; 
        
           }

My reasons for preferring this strategy instead of PyThread_get_thread_native_id() are:

It should be a lot faster than PyThread_get_thread_native_id() (no syscalls), although still not as fast as the platform-specific implementations currently used in _PyThread_Id()
We can make more assumptions about the format (i.e., non-zero, pointer-aligned), which I think will come in handy when implementing recursive mutexes.

You may have a look at _PyThreadState_GET() which is implemented by reading a thread local variable. If Py_BUILD_CORE_MODULE macro is defined, the function is implemented with an opaque function call: implementation for libraries.

Got it, I will send a PR soon.

I am going to submit the PR early next week :)

pythongh-112535: Implement fallback implementation of _Py_ThreadId()

gh-112535: Update _Py_ThreadId() to support RISC-V (gh-113084)

Merge remote-tracking branch 'upstream/main' into pythongh-112535-fal…

pythongh-112535: Update _Py_ThreadId() to support RISC-V (pythongh-11…

pythongh-112535: Implement fallback implementation of _Py_ThreadId()

gh-112535: Implement fallback implementation of _Py_ThreadId() (gh-11…

@colesbury

Okay, let's close the issue since the fallback implementation is merged.
Thank you for all your supports and contribution @colesbury, @vstinner(Awesome test codes!) and @furkanonder (RISC-V)

Now, we can support free-threaded CPython for all platforms where the fallback implementation(it only requires thread-local feature) works!

pythongh-112535: Implement fallback implementation of _Py_ThreadId() (p…

pythongh-112535: Add test on _Py_ThreadId() (python#112709)

pythongh-112535: Update _Py_ThreadId() to support PowerPC (pythongh-1…

pythongh-112535: Add comment for ppc32/64 registers (pythongh-112746)

pythongh-112535: Update _Py_ThreadId() to support s390/s390x (pythong…

pythongh-112535: Update _Py_ThreadId() to support RISC-V (pythongh-11…

pythongh-112535: Implement fallback implementation of _Py_ThreadId() (p…

Don't build the freethreding packages on ppc64le/s390x for now

pythongh-112535: Add test on _Py_ThreadId() (python#112709)

pythongh-112535: Update _Py_ThreadId() to support PowerPC (pythongh-1…

pythongh-112535: Add comment for ppc32/64 registers (pythongh-112746)

pythongh-112535: Update _Py_ThreadId() to support s390/s390x (pythong…

pythongh-112535: Update _Py_ThreadId() to support RISC-V (pythongh-11…

pythongh-112535: Implement fallback implementation of _Py_ThreadId() (p…

Navigation Menu

Repository navigation

Make `_Py_ThreadId()` work on PowerPC, IBM Z, etc. #112535

Feature or enhancement

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	#ifdef __APPLE__
	uint64_t native_id;
	(void) pthread_threadid_np(NULL, &native_id);
	#elif defined(__linux__)
	pid_t native_id;
	native_id = syscall(SYS_gettid);
	#elif defined(__FreeBSD__)
	int native_id;
	native_id = pthread_getthreadid_np();
	#elif defined(__FreeBSD_kernel__)
	long native_id;
	syscall(SYS_thr_self, &native_id);
	#elif defined(__OpenBSD__)
	pid_t native_id;
	native_id = getthrid();
	#elif defined(_AIX)
	tid_t native_id;
	native_id = thread_self();
	#elif defined(__NetBSD__)
	lwpid_t native_id;
	native_id = _lwp_self();
	#elif defined(__DragonFly__)
	lwpid_t native_id;
	native_id = lwp_gettid();
	#endif
	return (unsigned long) native_id;
	}

	static inline mi_threadid_t _mi_prim_thread_id(void) mi_attr_noexcept {
	return (uintptr_t)&_mi_heap_default;
	}

Make _Py_ThreadId() work on PowerPC, IBM Z, etc. #112535

Description

Feature or enhancement

Linked PRs

Activity

corona10 commented on Dec 8, 2023

colesbury commented on Dec 8, 2023

corona10 commented on Dec 8, 2023

colesbury commented on Dec 8, 2023

corona10 commented on Dec 10, 2023

colesbury commented on Dec 10, 2023

corona10 commented on Dec 10, 2023

colesbury commented on Dec 10, 2023

vstinner commented on Dec 11, 2023

corona10 commented on Dec 14, 2023

corona10 commented on Dec 18, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Make `_Py_ThreadId()` work on PowerPC, IBM Z, etc. #112535