<vstinner>
Py_TYPE() was one of last macro that I didn't convert, in my list of "most important C API functions"
<vstinner>
i already pushed the change in Python 3.10, but I had to revert it, since it broke too many C extensions. in the meanwhile, I created the pythoncapi_compat.h project which provides Py_SET_TYPE(obj, type) to Python 2.7-3.11, and I updated many C extensions
Atque has quit [Quit: ...]
<LarstiQ>
I remember that previous breaking-too-many-extensions.
<antocuni>
vstinner: I don't understand, what changed since the last time? I mean, if last time it broke too many extensions, why now would be better?
<vstinner>
sadly, "Py_TYPE(type) = base_type;" is commonly used when defining types statically in a C extensions, to workaround the stupid Windows MSC linker which is unable to get types like &PyLong_Type at build
<vstinner>
antocuni: i fixed many extensions and i created pythoncapi_compat
<antocuni>
cool
<vstinner>
antocuni: for example, Cython was updated and many "C extensions" are written with Cython
<vstinner>
most popular C extensions like Cython and numpy are now fixed
<vstinner>
antocuni: this incompatible C API change is annoying, but disallowing "Py_TYPE(obj) = type" hides a little bit more implementation details
<antocuni>
is there any particular reason why 4 are still broken? Is it just because you didn't have time to fix, or there are more fundamental reasons?
<vstinner>
antocuni: oh, i'm just too lazy to fix all projects :)
<antocuni>
ok :)
<vstinner>
antocuni: nah, it's trivial to fix a project. run my script, create a PR, get the PR merged
<vstinner>
antocuni: my upgrade_pythoncapi.py replaces "Py_TYPE(obj) = type" with "Py_SET_TYPE(obj, type)" for you, it's now automated
<antocuni>
👍
<vstinner>
i'm not sure if disallowing "Py_TYPE(obj) = type" solves any *major* issue for PyPy, but it should help to hide implementation details a little bit more
<vstinner>
as i wrote, the common usage was to define types statically
<vstinner>
but "Py_SIZE(obj) = size" seems common for some optimizations, i saw it in Cython
<antocuni>
I don't think it changes much w.r.t. to pypy
<vstinner>
antocuni: making PyObject opaque would be nice, but it seems hard. there are still many open issues
<vstinner>
antocuni: but i would be very happy if i can make PyTypeObject opaque. this structure changed a lot between Python versions, and it's bad that the structure size leaks in the ABI. at least, the structure is opaque in the stable ABI
<mattip>
vstinner: doesn't changing a macro into a function break the stable API?
<mattip>
It definitely changes the stable ABI, so what does that mean for abi3 c-extensions?
<mattip>
like cryptography
<vstinner>
mattip: there are two things: the "limited C API" and the "stable ABI". changing Py_TYPE() doesn't change anything to the stable ABI. but it breaks the limited C API
<vstinner>
mattip: do you think that it has an impact on the ABI?
<mattip>
yes, the headers now change, so if I compile cryptography on python 3.11 and target python 3.7, it won't work
<vstinner>
mattip: why would it break?
<vstinner>
mattip: maybe i miss something
<vstinner>
mattip: in 3.11, Py_TYPE() is declared as a static inline function which reads PyObject.ob_type (ob->ob_type)
<vstinner>
mattip: at the ABI level, it should be the same
<mattip>
ahh, it is static in the headers, not a function exported from libpython?
<vstinner>
mattip: maybe the compiler doesn't want to inline, creates a Py_TYPE() function in your C extension, and then calls it
<vstinner>
mattip: in this case, the function is part of your C extension
<mattip>
so there is no change at all and this is just syntactical churn
<vstinner>
mattip: the *API* change is that "Py_TYPE(obj) = type" is now a compiler error
<mattip>
isn't that valid if done before PyTypeReady?
<vstinner>
mattip: C extensions use it *before* calling PyType_Read() to work around compiler/linker limitation
<vstinner>
mattip: some linkers (Windows?) disallows to specific ".tp_type = &PyLong_Type," statically, they require to do it "at runtime" (in the module init code) using "Py_TYPE(MyType) = &PyLong_Type;"
<vstinner>
to specify*
<mattip>
I don't see what problem this is solving, it does not make the ob_type field more opaque
<vstinner>
mattip: you are making the assumption that PyObject.ob_type exists. i would like to make PyObject fully opaque
<mattip>
you can't
<mattip>
you cannot even get ob_type out of the header files
<vstinner>
mattip: yes i can and i will :-D
<mattip>
not until you break abi3
<vstinner>
mattip: i'm open to break everything :-D
<vstinner>
mattip: so far, my work didn't touch the stable ABI. i prepared the code to make it possible
<vstinner>
i'm trying to fix the *API* for now
<mattip>
I don't think this is appropriate for #pypy
<vstinner>
mattip: i think we already had this conversation 2 or 3 times :)
<vstinner>
mattip: i announced my change here because PyPy has pycext
<mattip>
is there a concrete change we must make to cpyext?
<vstinner>
mattip: i don't know cpyext, i cannot say
<mattip>
is there a test we should be passing that currently we fail?
<vstinner>
i guess that it should support Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE() added to Python 3.8
<mattip>
I don't see any tests for those in _testcapimodule.c
<vstinner>
mattip: the C API is badly tested in CPython. _testcapi is not desigend to test the C API. it's a place were we put anything used by unit tests which cannot be done in Python
<mattip>
cool, so we will hit that when we get to python 3.10. We are currently releasing pypy for python 3.8
<vstinner>
oh sorry, Py_SET_{REFCNT,TYPE,SIZE} functions were added to Python 3.9 (not 3.8)
<mattip>
hopefully by then HPy will be a viable alternative to all this
<vstinner>
sadly, i don't think that it's going to happen soon :-(
<vstinner>
i mean, even if it's a viable alternative, existing projects are not going to be fully converted to HPy soon
<mattip>
why not? Once cython/pybind11 have support, other projects can follow
<mattip>
just as they can use your compatibility shim
<vstinner>
mattip: well. in practice, sadly, there are still projects written directly with the C API. see https://bugs.python.org/issue39573#msg401378 for the example of the Py_TYPE() case
<vstinner>
i also hope that Cython and HPy will be used widely, but before, it's the case, IMO we can do "something" :-)
<mattip>
ok. I hope it goes well.
<vstinner>
mattip: people will complain, i'm used to it :-D
<mattip>
well, after enough people told me numpypy was not going to work I dropped it
<vstinner>
:-(
<mattip>
if you are alone in the struggle, you might want to rethink your approach
<vstinner>
mattip: i have to do. it's a tedious topic. it's hard to explain the rationale in a few sentences. i'm confident that it's worth it, and i already pushed changes in Python 3.8 & 3.9 which went fine ;-)