<Hodgestar>
For HPy we need tp_traverse to be called for every HPyField.
<Hodgestar>
Sorry, the visit function needs to be called within tp_traverse for every HPyField.
<Hodgestar>
CPython seems to be lax in ways that are specific to its own garbage collection (and to the cyclic garbage collector having been bolted on later) and HPy will likely have to have more consistent rules to cater to a wider class of garbage collectors (i.e. to not expose the inner workings of the current CPython garbage collector).
<Hodgestar>
I don't think this is a bad thing -- hiding the inner workings of the interpreter is one of the core goals of HPy.
<Hodgestar>
From our discuss, I liked Antonio's suggestion to store the information needed in tp_traverse for the more complex numpy dtypes on the array instance so that tp_traverse can get at it easily.
<mattip>
so we need to get some kind of buy in from CPython to change the requirements of tp_traverse, something like
<mattip>
"tp_traverse should not require C-API calls to find the owned objects to visit"
<Hodgestar>
Well, first we need CPython to require tp_traverse to visit all objects and not just the ones that could format part of reference cycles.
<Hodgestar>
I guess first we should see what real tp_traverse calls in the wild do -- i.e. are there a lot of complex cases like numpy, or only a few?
<Hodgestar>
And how common is it to just leave out tp_traverse in custom extension types that extend the PyObject struct in ways that reference other Python objects that happen to only be integers, strings, etc that one would not have to call tp_traverse for in CPython.
<mattip>
numpy only actually uses tp_traverse in dtype metaclasesses
<mattip>
and in both cases I don't think these will break any cycles
<mattip>
OTOH cython uses it
mattip has quit [Ping timeout: 256 seconds]
mattip has joined #hpy
<antocuni_>
mattip: I don't understand why we need CPython to change the requirements of tp_traverse. There are already many things that you can do in CPython but not in HPy, and calling C-API functions from within tp_traverse is just another of those
<antocuni_>
of course this means that for some use cases it will be harder to rewrite tp_traverse to be HPy-compatible, but I don't think there is much that we can do for that
<antocuni_>
also, sorry for having been silent in my last 15 minutes of the call, my mic was turned off and you didn't hear anything of what I said (maybe it's not that bad 😅)
<antocuni_>
did I miss anything important after I left?
<mattip>
I also left before the end
<mattip>
well, when we come to ask projects to rewrite for HPy, it would be nice to have at least a "please don't do that" in the CPython documentation
<Hodgestar>
And definitely the HPy documentation. :)
<fangerer>
> did I miss anything important after I left?
<fangerer>
I don't want to judge that :slightly_smiling_face: . We basically continued the discussion of problems when mixing HPyFields with other (not managed) C data. Ronan also pointed out that Numpy uses memcpy to copy data that may contain Python objects and C primitives. We do currently not explicitly state in the docs that you may not do that (in general). Although, it will most certainly work if the copied fields are again owned by the
<fangerer>
same object (useful in case you are resizing a C array). I think we need to extend our docs a bit.
<steve_s>
> if the copied fields are again owned by the same object
<steve_s>
that assumption may not work for all types of GCs, so imo it would be better to avoid this too. In general moving memory managed by GC behind GC's back is not great.
antocuni_ is now known as antocuni
<antocuni>
indeed, that's a tricky case
<steve_s>
There was a question whether mixing native and managed memory is a good idea after all, but anything else would make the migration more complicated...
<antocuni>
for example, what is you want to implement a list-like object in C? You surely need to resize the underlying array, and you want memcpy or something similar to copy the contents to the new memory
<antocuni>
you surely don't want to be forced to copy the items one by one
Techcable has quit [Ping timeout: 240 seconds]
<steve_s>
would it be hard to dismarry the native parts and the HPyFields? Certainly not great for cache locality I know... Ok, we may have to allow moving memory potentially containing HPy fields within one owning object.
<antocuni>
what do you mean exactly?
<steve_s>
Have a region of memory that contains only HPyFields and memory that contains everything else. You could memcpy/move at least the other region
<antocuni>
and what do I do if I want a struct Point {int x; int y; HPyField name;} ?
<steve_s>
if you have that struct in an array you would have one array of struct Point {int x; int y; } and one array of HPyField name
<antocuni>
and how would it help?
<antocuni>
I mean, I'm not sure to understand what is the problem that you are trying to solve
<steve_s>
there would be HPyField_ArrayMove or something like that that would allow the GC to see that things are moving around. The problem, in general, is the question: if you memmove some memory that contains HPyFields, should you then call HPyField_Store? I think that would defeat the purpose of memmove. So we may have to specify that moving memory around within one object is fine, which seems bit dangerous to me, but maybe I am too cautious
<antocuni>
you cannot use HPyField_Store because that one receives an HPy, not an HPyField
<antocuni>
I think the core of your question is:
<steve_s>
ah righ, so you'd have to load them also
<antocuni>
how does the GC knows which are the GC-managed pointers? Does it know by keeping track of HPyField_Store, or by calling tp_traverse?
<antocuni>
I think that so far the answer is "by calling tp_traverse"
<antocuni>
i.e., memcpy-ing memory around is fine as long as tp_traverse does the right thing (and you must be extra cautious to now invoke the GC while the copy is in progress)
<antocuni>
i.e., the following will lead to problems:
<antocuni>
HPyField *tmp = mylist->items;
<steve_s>
in graalpython we need to keep them separate, because we cannot store them into unmanaged memory and have GC understand them. Another way of looking at this: when I call HPyField_Store(ctx, owner, &owner->foo->bar, value) should the HPyField data stay at the address &owner->foo->bar
<antocuni>
HPyField *new = malloc(...);
<antocuni>
memcpy(new, tmp)
<antocuni>
foo()
<antocuni>
mylist->items = new;
<antocuni>
if foo() invokes the GC, then things might explode because tp_traverse return the old array
<antocuni>
but apart for that, if foo() does "nothing", the code above should be fine
<steve_s>
yes, atomicity is another issue
<antocuni>
steve_s: ah, so from the graalpython point of view, tp_traverse() is just useless?
<steve_s>
still with this example you have the HPyField in two places suddenly and the Python engine doesn't know about that. Could it be an issue for future optimizations/new GCs/etc. I suspect?
<antocuni>
"in two places"? You mean tmp and new?
<steve_s>
yes. The best would be if the specs said that tp_traverse must visit all HPyFields and it is up to the Python implementation whether it keeps track of them internally, or if it takes advantage of tp_traverse
<steve_s>
> ah, so from the graalpython point of view, tp_traverse() is just useless?
<antocuni>
ok, I understand the problem now
<antocuni>
and I see why memcpy is a problem
<antocuni>
one possibile solution could be that after the memcpy (or equivalent) the C code must call "HPy_UpdateGC(ctx, mylist)" or something similar
<antocuni>
which would do nothing on e.g. pypy
<antocuni>
on graalpython, you can use it as a hook where to call tp_traverse and update your internal tracking?
<antocuni>
I don't know whether it would work
<steve_s>
I think it can, but not sure how efficient. To be clear I think that memcpy/memmove is ok right now if it does not change the owner of the field (also for GraalPython). Forward looking I wonder if this isn't still dangerous w.r.t. enabling future improvements. Also the property of HPyField_Store(ctx, owner, &owner->foo->bar, value) => I can assume the field data stays at the address &owner->foo->bar does not hold. Again, not useful today
<antocuni>
yes, I see
<steve_s>
Another property that may change behind Python's back is that there is one copy of the field, if you can copy them around. Not useful today, can it be useful in the future?
<antocuni>
the problem with the two-arrays solution that you propose is that e.g. if fundamentally incompatible with numpy structured arrays, where it's the user to decide what is the content of the strcut
<steve_s>
right, ok, then that's no go then
<antocuni>
but I think that finding an API which works well with both the "pypy approach" and the "graalpython approach" is important. Abstracting over the implementation details is the core goal of hpy, after all
<antocuni>
I don't know how, though :)
Techcable has joined #hpy
<arigato>
just giving my 2 cents here, in my own opinion: debating basic issues like that after more than 2 years?? sorry, but hpy is not going anywhere IMHO
<antocuni>
arigato: not a very helpful comment, admittedly