<Hodgestar>
It's good to see lots of buzz around HPy again.
<cfbolz>
Hodgestar: any new buzz?
<Hodgestar>
Mostly Guido mentioning HPy as the way forward in a python-dev thread, the Cython and numpy work coming along, and the piconumpy performance discussion thread.
<Hodgestar>
steve_s: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_iternext says that tp_iternext can signal the end of a iterator by returning NULL without setting a StopIteration exception, so maybe this can be made better?
<Hodgestar>
steve_s: If this were NOT already the case, I think this would be a situation in which HPy might change the definition of tp_iternext to not require the exception to be set and to create one in the CPython HPy if needed -- i.e. we don't have to follow CPython where it seems like a bad idea.
<steve_s>
Ah, that looks like what we'd need thanks. Yes, I was thinking and we may want to change the contract a bit for such situations. Some things may be "mostly" compatible: like you can still set StopIteration if you want.
<Hodgestar>
Woot. Yes, in this case I think what is already there is okay. Maybe we should just document that setting StopIteration might be a slow and rather cumbersome way to do things from C.
<Hodgestar>
antocuni: I wonder if it is time to start thinking about the "HPy Argument Clinic" idea again -- i.e. exposing the underlying C methods directly to avoid all the argument wrapping. We don't necessarily have to do it right now, but it seems like it would be good to have a plan that people working could try to follow.
<antocuni>
yes, eventually we want it
<steve_s>
do you have a plan how it would be done technically? E.g., some python package that would preprocess the C sources, some macro magic, ...?
<antocuni>
I don't know
<antocuni>
CPython uses a preprocessor; you mark your C code with special comments and run the preprocessor
<antocuni>
the preprocessor modifies the file in place, in such a way that you can still edit the non-clinic parts of the code and rerun it later
<antocuni>
so, this is surely one of the options on the table
<Hodgestar>
steve_s: Probably the first step is figuring out what the result after the processing should look like. I.e. what information has the Python implementation need in order for it to be useful & maybe for functions to call each other without wrapping all their C constants. Then we can think about how to get there.
<antocuni>
another option is try to use C macros, but I don't know how feasible it is
<Hodgestar>
s/constants/variables/
<antocuni>
I think we also need to list explicitly what are our goals
<antocuni>
one goal is to make it easier to expose C functions to Python, similar to what CFFI does
<antocuni>
but another goal is to allow interpreters to call directly the underlying C function if they want
<steve_s>
thinking about it more: HPy code should probably just indicate the calling convention to the Python engine and the Python engine would then be responsible for the actual conversion? So PyPy and GraalPython con JIT/optimize that
<antocuni>
which means we need to store info about the signature somewhere
<antocuni>
yes, exactly
<antocuni>
but we still have the problem of CPython, for which we need to generate C code statically
<antocuni>
another possible use case is Cython
<steve_s>
It can be just BuildValue/ArgParse type of string: e.g., "ii" for two long arguments?
<antocuni>
if Cython statically knows that I'm calling foo() with (int64, int64), it might generate a fast-path case for that. But then it means that it needs to be able to quickly check at runtime whether foo() supports (int64, int64)
<antocuni>
one of the ideas that have been floating in my mind is to encode the signature in a 64 bit integer. We have only 4 types: void, int64, float64 and HPy (i.e. python objects)
<antocuni>
so each type can be encoded in 2 bits
<antocuni>
in 64 bit, we can encode a signature of up to 31 arguments (+1 return type)
<steve_s>
are we now talking about upcalling to Python or downcalling from Python to some HPy function?
<antocuni>
which means that it will be very quick e.g. for cython to check the fast path I was talking about
<antocuni>
what is upcalling and downcalling?
<steve_s>
(ah, that's probably just our terminology) up == C -> Python, down Python -> C
<antocuni>
I'm talking about both
<antocuni>
support I write my function int64 foo(int64, int64)
<antocuni>
I want to generate a wrapper which makes it possible to call foo from Python
<antocuni>
but I also want to be able to call foo from e.g. Cython-generated C code
<antocuni>
or JIT compiled PyPy/GraalPython code
<cfbolz>
antocuni: that's even independent of hpy, really
<cfbolz>
cython has this external function and could expose it
<antocuni>
cfbolz: I think that you are talking about Cython-generated code
<antocuni>
I mean, calling a Cython-generated function
<antocuni>
I'm talking about cython calling an HPy function which could be written manually in C
<cfbolz>
Yep
<cfbolz>
Fine
<cfbolz>
Just saying even with no hpy in sight anywhere this would be a very useful feature for the cython+jit combination
<antocuni>
ah ok, in that sense yes
<antocuni>
I have a vague remembering that there was a PEP to expose the underyling C function of a Python function, but I don't remember the details
<Hodgestar>
It would be nice if ordinary C libraries could signal this too somehow, although the options there are quite a bit more limited (and I have no idea what one would do for more complex types -- I guess one might end up with something like CFFI?)
<cfbolz>
Hodgestar: that doesn't help performance though
<cfbolz>
For cython it helps because of wrapping/unwrapping
<Hodgestar>
Could one imagine that Cython might know something about CFFI wrappers and call the underlying functions directly? I'm guessing not because of what the generated C modules generated by CFFI look like?
<Hodgestar>
I do not know the internals of CFFI.
<antocuni>
Hodgestar: at the moment the CPython C API doesn't provide a way to get the underlying function so no
<antocuni>
but if the C API and/or HPy provides a way to do it then yes, cython could in theory use it
<Hodgestar>
Woot.
<vstinner>
antocuni: there is a lot of on-going work to optimize function calls in Python/ceval.c