<vstinner>
do you think that this statement is correct?
<vstinner>
i know that this PEP alone will not magically convert existing C extensions to HPy. the question is if you think that it's correct that replacing "Py_SIZE(obj) = new_size" with "Py_SET_SIZE(obj, size)" helps HPy
<vstinner>
"If a project is portrayed as benefiting from or supporting the PEP, make sure there is some direct indication from the project included to make the support clear. This is to avoid a PEP accidentally portraying a project as supporting a PEP when in fact the support is based on conjecture."
<antocuni>
I don't think it helps much concretely. If I am porting a module to HPy I have to migrate C-API calls anyway
<Hodgestar>
If you feel like annoying people about deprecations, you could ask that __str__ be added back to int because it was removed without deprecation in 3.8. :P
<vstinner>
antocuni: "I have to migrate C-API calls anyway" sure, HPy requires adding a "ctx" argument. but is it simpler to replace Py_SET_SIZE(obj, new_size) with HPy_SET_SIZE(ctx, obj, new_size), than replaceing "Py_SIZE(obj) = new_size" with "HPy_SET_SIZE(ctx, obj, new_size)"?
<Hodgestar>
+1 on antocuni's reply, but also agreed that the fewer strange uses of macros the easier things are likely to be for everyone maintaining C-APIs (including CPython).
<Hodgestar>
And probably searching and replacing Py_SET_SIZE is easier than searching and replacing some strange macro uses of Py_SIZE.
<vstinner>
Hodgestar: do you think that "Disallowing the usage of macros as l-value helps the migration of existing C extensions to HPy by reducing differences between the C API and the HPy API." should be rephrased, or does it look good to you?
<vstinner>
Hodgestar, antocuni: or do you strongly disagree and prefer that i don't mention HPy at all?
<phlebas>
I would say the automatic search-and-replace definitely feels safer for "Py_SET_SIZE(obj, size)" than it would for the macro
<phlebas>
the regex should also be more sane, I would think
<vstinner>
phlebas: regex are "easy" to write, but there are multiple traps :) i tried to manually avoid them
<vstinner>
for example, "return frame->f_code;" is replaced with "return _PyFrame_GetCodeBorrow(frame);" whereas "frame->f_code = code;" is left unchanged: "_PyFrame_GetCodeBorrow(frame) = code;" fails with a compilation error
<vstinner>
i considered to use coccinelle, but i like the idea of only requiring Python :) Python is easy to install on most platforms
<antocuni>
I agree that that porting "Py_SET_SIZE" to HPy is marginally easier, but only very marginally IMHO
<antocuni>
that said, for the specific case of Py_SIZE, I don't think there will ever be an HPy equivalent 😅
<vstinner>
antocuni: Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE() are really CPython specific :-/
<antocuni>
it would be probably easier to discuss the merit of something which *does* have a hpy equivalent
<vstinner>
antocuni: i chose to add them to the limited C API, rather than the CPython specific C API mostly because of the backward compatibility
<antocuni>
even PyTuple_GET_ITEM doesn't apply to HPy because to build tuples you use HPyTupleBuilder
<vstinner>
the PEP 674 makes the C API less insane. today, it's technically valid to do "PyFloat_AS_DOUBLE(number) = 1.0;": modify an immutable float number
<vstinner>
i don't think that anyone want to support that, even in CPython
<antocuni>
I perfectly agree with the PEP and I think it should be accepted
<antocuni>
I'm just replying to the question "does it help HPy?"
<vstinner>
antocuni: i cannot modify PyTuple_GET_ITEM() in this PEP because i want to continue supporting the other insane code: super_fast_array = &PyTuple_GET_ITEM(tuple, 0) which is *commonly* used by C extensions
<antocuni>
yes I know, it's also used by hpy itself :)
<antocuni>
(in some cpython-specific part of the code)
<vstinner>
"yes I know, it's also used by hpy itself :)" using that internally is perfectly fine. exposing it in the public C API is... bad :-(
<vstinner>
antocuni: i'm thinking about adding 2 APIs to access arrays of PyObject* and byte strings, to have like acquire/release resource functions, similar to PyBuffer_Release()
<antocuni>
this looks very similar to what HPy does
<vstinner>
so the string can be copied somewhere and the memory is released when the resource is released
<vstinner>
or the memory can be pinned, depending on the object type and the object size
<vstinner>
antocuni: PyBytes_AS_STRING() is not compatible with objects moving in memory :-(
<antocuni>
see also https://github.com/hpyproject/hpy/pull/259 which goes even further and mprotect() the memory after it's no longer valid (only in debug mode, of course)
<vstinner>
antocuni: that's a super cool debug feature! :)
<antocuni>
thanks to steve_s for implementing it
<vstinner>
antocuni, Hodgestar : Hodgestar wrote "but also agreed that the fewer strange uses of macros the easier things are likely to be for everyone maintaining C-APIs" so sorry, i'm not sure if i should removed the whole HPy section or not
<vstinner>
antocuni: i let you go through the list of modified macros to see if any would benefit to HPy. but i don't see a clear candidate, since only 2 clearly need a "SET" function, and these 2 are very specific to CPyhon: Py_SET_SIZE() and Py_SET_TYPE()
<Hodgestar>
vstinner: The fewer internal details exposed via macros, the easier it will be for HPy to provide direct equivalents. Any macro that references "non-public" interfaces effectively exposes those interfaces publicly.
<Hodgestar>
vstinner: Maybe the same is technically true for functions that are always inlined, but at least people would have to type ".ob_refcount" themselves (for example) and not have it magically typed for them by an innocuous looking macro.
<vstinner>
Hodgestar: the PEP 674 is limited to the API. each Python implementation is free to decide how it's implemented at the ABI level ;-)
<vstinner>
Hodgestar: my (draft) PEP 670 gives the ability to convert macros to regular functions to hide implementation details \o/
<vstinner>
antocuni, phlebas, Hodgestar: i tried to summarize and rephase what you wrote. tell me if you prefer to not be cited in this "endorsement" :-) https://github.com/python/peps/pull/2169
<antocuni>
+1 for me
<antocuni>
although you should probably specify that Py_SET_SIZE is just an example because this is not going to have an HPy equivalent
<vstinner>
antocuni: i updated the PR to remove the PyPy section since mattip is not convinced that this PEP alone helps to avoid creating CPython objects in cpyext
<vstinner>
i wanted to mention RustPython but I checked and it simply has no C API support at all right now. they consider supporting HPy
<vstinner>
also, GraalPython supports HPy, but it not going to support the C API neither, if i understood correctly
<antocuni>
I think that GraalPython DOES support the C API already
<vstinner>
antocuni: the doc says "The GraalVM team is actively working to enable support for the Python C API"
<antocuni>
I'll let the graalpython people to answer since they obviously know more than me :)
<vstinner>
antocuni: i don't know who should I contact for GraalPython
<antocuni>
fangerer, phlebas and steve_s
<vstinner>
fangerer, phlebas, steve_s: hi. do you think that GraalPython would benefit of https://www.python.org/dev/peps/pep-0674/ ? if yes, can you write a short paragraph that I can simply copy/paste in the PEP? :-)
<phlebas>
we do, but it's expensive for us. mainly because we need to emulate all the internal structs if they are exposed. it definitely helps us to reduce macros, since we ideally have everything as functions so we can as much as possible treat all PyObject structs as opaque