cfbolz changed the topic of #pypy to: #pypy PyPy, the flexible snake | IRC logs: and | insert pithy quote here
fotis has joined #pypy
fotis has quit [Ping timeout: 260 seconds]
lritter has quit [Ping timeout: 240 seconds]
lritter has joined #pypy
Atque has quit [Quit: ...]
Atque has joined #pypy
yuiza has joined #pypy
yuiza has quit [Remote host closed the connection]
Corbin has quit [Ping timeout: 245 seconds]
otisolsen70 has joined #pypy
stkrdknmibalz has quit [Quit: WeeChat 3.0.1]
<vstinner> hi. for the 3rd time (!), I modified the C API Py_TYPE() macro to disallow "Py_TYPE(obj) = type;" syntax. I converted the macro to a static inline function:
<vstinner> Py_TYPE() was one of last macro that I didn't convert, in my list of "most important C API functions"
<vstinner> i already pushed the change in Python 3.10, but I had to revert it, since it broke too many C extensions. in the meanwhile, I created the pythoncapi_compat.h project which provides Py_SET_TYPE(obj, type) to Python 2.7-3.11, and I updated many C extensions
Atque has quit [Quit: ...]
<LarstiQ> I remember that previous breaking-too-many-extensions.
<antocuni> vstinner: I don't understand, what changed since the last time? I mean, if last time it broke too many extensions, why now would be better?
<vstinner> sadly, "Py_TYPE(type) = base_type;" is commonly used when defining types statically in a C extensions, to workaround the stupid Windows MSC linker which is unable to get types like &PyLong_Type at build
<vstinner> antocuni: i fixed many extensions and i created pythoncapi_compat
<antocuni> cool
<vstinner> antocuni: for example, Cython was updated and many "C extensions" are written with Cython
<vstinner> antocuni: 11 are fixed, 1 has a pending fix, 4 are broken:
<vstinner> most popular C extensions like Cython and numpy are now fixed
<vstinner> antocuni: this incompatible C API change is annoying, but disallowing "Py_TYPE(obj) = type" hides a little bit more implementation details
<antocuni> is there any particular reason why 4 are still broken? Is it just because you didn't have time to fix, or there are more fundamental reasons?
<vstinner> antocuni: oh, i'm just too lazy to fix all projects :)
<antocuni> ok :)
<vstinner> antocuni: nah, it's trivial to fix a project. run my script, create a PR, get the PR merged
<vstinner> antocuni: my replaces "Py_TYPE(obj) = type" with "Py_SET_TYPE(obj, type)" for you, it's now automated
<antocuni> 👍
<vstinner> i'm not sure if disallowing "Py_TYPE(obj) = type" solves any *major* issue for PyPy, but it should help to hide implementation details a little bit more
<vstinner> as i wrote, the common usage was to define types statically
<vstinner> but "Py_SIZE(obj) = size" seems common for some optimizations, i saw it in Cython
<antocuni> I don't think it changes much w.r.t. to pypy
<vstinner> antocuni: making PyObject opaque would be nice, but it seems hard. there are still many open issues
<vstinner> antocuni: but i would be very happy if i can make PyTypeObject opaque. this structure changed a lot between Python versions, and it's bad that the structure size leaks in the ABI. at least, the structure is opaque in the stable ABI
<mattip> vstinner: doesn't changing a macro into a function break the stable API?
<mattip> It definitely changes the stable ABI, so what does that mean for abi3 c-extensions?
<mattip> like cryptography
<vstinner> mattip: there are two things: the "limited C API" and the "stable ABI". changing Py_TYPE() doesn't change anything to the stable ABI. but it breaks the limited C API
<vstinner> mattip: do you think that it has an impact on the ABI?
<mattip> yes, the headers now change, so if I compile cryptography on python 3.11 and target python 3.7, it won't work
<vstinner> mattip: why would it break?
<vstinner> mattip: maybe i miss something
<vstinner> mattip: in 3.11, Py_TYPE() is declared as a static inline function which reads PyObject.ob_type (ob->ob_type)
<vstinner> mattip: at the ABI level, it should be the same
<mattip> ahh, it is static in the headers, not a function exported from libpython?
<vstinner> mattip: maybe the compiler doesn't want to inline, creates a Py_TYPE() function in your C extension, and then calls it
<vstinner> mattip: in this case, the function is part of your C extension
<mattip> so there is no change at all and this is just syntactical churn
<vstinner> mattip: the *API* change is that "Py_TYPE(obj) = type" is now a compiler error
<mattip> isn't that valid if done before PyTypeReady?
<vstinner> mattip: C extensions use it *before* calling PyType_Read() to work around compiler/linker limitation
<vstinner> mattip: some linkers (Windows?) disallows to specific ".tp_type = &PyLong_Type," statically, they require to do it "at runtime" (in the module init code) using "Py_TYPE(MyType) = &PyLong_Type;"
<vstinner> to specify*
<mattip> I don't see what problem this is solving, it does not make the ob_type field more opaque
<vstinner> mattip: the rationale is explained in and
<vstinner> mattip: you are making the assumption that PyObject.ob_type exists. i would like to make PyObject fully opaque
<mattip> you can't
<mattip> you cannot even get ob_type out of the header files
<vstinner> mattip: yes i can and i will :-D
<mattip> not until you break abi3
<vstinner> mattip: i'm open to break everything :-D
<vstinner> mattip: so far, my work didn't touch the stable ABI. i prepared the code to make it possible
<vstinner> i'm trying to fix the *API* for now
<mattip> I don't think this is appropriate for #pypy
<vstinner> mattip: i think we already had this conversation 2 or 3 times :)
<vstinner> mattip: i announced my change here because PyPy has pycext
<mattip> is there a concrete change we must make to cpyext?
<vstinner> mattip: i don't know cpyext, i cannot say
<mattip> is there a test we should be passing that currently we fail?
<vstinner> i guess that it should support Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE() added to Python 3.8
<mattip> I don't see any tests for those in _testcapimodule.c
<vstinner> mattip: the C API is badly tested in CPython. _testcapi is not desigend to test the C API. it's a place were we put anything used by unit tests which cannot be done in Python
<vstinner> mattip: but if you was talking about CPython, Py_SET_TYPE() is now "tested" at:
<mattip> cool, so we will hit that when we get to python 3.10. We are currently releasing pypy for python 3.8
<vstinner> oh sorry, Py_SET_{REFCNT,TYPE,SIZE} functions were added to Python 3.9 (not 3.8)
<mattip> hopefully by then HPy will be a viable alternative to all this
<vstinner> sadly, i don't think that it's going to happen soon :-(
<vstinner> i mean, even if it's a viable alternative, existing projects are not going to be fully converted to HPy soon
<mattip> why not? Once cython/pybind11 have support, other projects can follow
<mattip> just as they can use your compatibility shim
<vstinner> mattip: well. in practice, sadly, there are still projects written directly with the C API. see for the example of the Py_TYPE() case
<vstinner> i also hope that Cython and HPy will be used widely, but before, it's the case, IMO we can do "something" :-)
<mattip> ok. I hope it goes well.
<vstinner> mattip: people will complain, i'm used to it :-D
<mattip> well, after enough people told me numpypy was not going to work I dropped it
<vstinner> :-(
<mattip> if you are alone in the struggle, you might want to rethink your approach
<vstinner> mattip: i have to do. it's a tedious topic. it's hard to explain the rationale in a few sentences. i'm confident that it's worth it, and i already pushed changes in Python 3.8 & 3.9 which went fine ;-)
<vstinner> mattip: my latest article on this topic:
<vstinner> mattip: bye ;-)
<mattip> again, good luck
_whitelogger has joined #pypy
otisolsen70 has quit [Quit: Leaving]
[Arfrever] has quit [Read error: Connection reset by peer]
[Arfrever] has joined #pypy
[Arfrever] has quit [Read error: Connection reset by peer]
[Arfrever] has joined #pypy