DuToit has quit [Read error: Connection reset by peer]
DuToit has joined #hpy
<arigato>
note: the 'gc' is not a GC-managed instance, because the class contains _alloc_flavor_='raw'
<antocuni>
ah yes, I noticed that it's a <Struct> instead of <GcStruct> but didn't investigate further
<antocuni>
I'm having a different problem now, though
<antocuni>
because I need to find a way to specialize _trace_one_field for the different callbacks
<arigato>
do things explode if you merge all the callbacks together in this case?
<arigato>
i.e. is it done only for performance, or also for correctness
<antocuni>
I have tried various combinations of @specialize.memo and @specialize.arg but they all explode for different reasons
<antocuni>
I think it is needed to specialize on the callback because it can be very different things
<arigato>
OK
<antocuni>
e.g., it can be _append_rpy_referent which is a global func, or BaseWalker.unobj which is a method
<arigato>
I would try a @specialize.memo that takes as argument the callback, and returns the ll function pointer you can pass to tp_traverse
<arigato>
ah, but yes, it won't work if callback is a bound method
<antocuni>
yes exactly, I tried that and it explodes with this:
<antocuni>
memo call with a class or PBC that has no corresponding Python object (<MethodDesc 'append' of <ClassDef 'targetnopstandalone.GCClass'> bound to <ClassDef 'targetnopstandalone.GCClass'> {}>)
<antocuni>
"append" is just a fake name, because I'm trying a minimal example
<antocuni>
arigato: do you know what are the possible callbacks? By grepping for gc.trace I found _append_rpy_referent, BaseWalker.unobj, MemoryPressureCounter._ref and HeapDumper.writeobj
<antocuni>
I wonder whether there are more
<antocuni>
and whether turning all callbacks from methods to normal functions can work
<arigato>
I think they could be functions with an extra 'arg' argument, if that would help
<antocuni>
yes, that's what I was thinknig
<antocuni>
then we need a way to turn 'arg' into a correctly typed BaseWalker
<antocuni>
(it's a target to translate with ./bin/rpython)
<antocuni>
if I uncomment ll_trace3, it works. With ll_trace3, it fails with "memo call with a class or PBC that has no corresponding Python object"
<antocuni>
I also tried to use @specialize.arg(0) instead of @memo for make_wrapper_with_callback, but in that case it fails with "RPython functions cannot create closures"
<arigato>
yes, the argument is not a constant
<arigato>
the argument to get_llhelper()
<antocuni>
I thought that specailized() arguments were considered constants, but probably not "constant enough"
<antocuni>
I admit I'm a bit confused by all the slightly different kind of specializations :)
<antocuni>
can you think of a different hack, or the only way is to turn all callbacks into functions?
<arigato>
the point is that BASE_WALKER.unobj can't be turned into a C function pointer
<antocuni>
which it might even be a good idea and a better interface, but as usual I'm a bit worried to modify the GC
<arigato>
because a C function pointer cannot store additional information like the instance BASE_WALKER
<antocuni>
right
<arigato>
yes, make them all functions with one extra argument, and pass None for the cases that are already functions now, and keep the @specialize; I think doing that should have no impact at all on performance
<arigato>
of course we really want to call tp_traverse with one "void*" argument
<antocuni>
what do you mean?
<arigato>
tp_traverse's signature is int tp_traverse(stuff, visitproc visit, void *arg);
<antocuni>
yes, but I don't understand how it's related
<arigato>
the idea is to have gc.trace(obj, callback, arg1, arg2), which calls your custom tracer, which calls tp_traverse(.., visit, arg), which calls visit(x, arg), which should then call callback(x, arg1, arg2)
<arigato>
so you need to pack arg1 and arg2 into something that can be passed as 'void* arg'
<arigato>
and this needs to be done carefully because arg1 and arg2 have various types depending on the specialization on 'callback'
<antocuni>
ah right, I see
<antocuni>
and moreover, packing arg1 and arg2 into a void* is a mess of its own
<antocuni>
because we cannot allocate and AFAIK we don't have any way to allocate some space on the stack in RPython
DuToit has quit [Remote host closed the connection]
DuToit has joined #hpy
<antocuni>
arigato: btw, arg2 might not be necessary, because all the method-callbacks which I found so far pass None as arg1
<arigato>
...ok
<antocuni>
a completely different solution could be to call tp_traverse with a callback which saves the pointers somewhere, and then iterate over all the pointers to call the gc-provided callback
<antocuni>
but I cannot imagine how to do that without putting a maximum limit to the number of pointers visited by tp_traverse
<arigato>
yes, bad idea
<arigato>
at this point you should just use rffi.cast(), too
<arigato>
not try to get around with the 27 other versions
<antocuni>
rffi.cast to pass around "arg"?
<arigato>
yes
<antocuni>
yes, but my problem is how to cast it back
<arigato>
that's a mess because you need class names and stuff and it only works with instances etc.
<antocuni>
but I need it to call e.g. BaseWalker.unobj()
<arigato>
right, at this point if I was doing it I would say "too bad it only works after translation"
<antocuni>
ah yes, I have already lost the hopes of testing it before translation
<antocuni>
if we find a way to make it working after translation, I'm already happy
<arigato>
it's a mess anyway, of course
<arigato>
sorry, I see the problem but don't have a solution so far
<antocuni>
no problem
<antocuni>
knowing that it's a problem without an easy solution is already something, at least I can avoid wasting time with the various hacks which we discussed
<antocuni>
ah, maybe one very hacky solution is to store the instance in a global
<antocuni>
can the trace functions be called concurrently?
<arigato>
it probably works
<arigato>
it should be all GIL-protected
<cfbolz>
btw, I was wondering whether there are restrictions on what you can and can't do in tp_traverse C functions
<cfbolz>
can't the hpyfields be in a bit of a weird state because some of them might have been forwarded already?
<antocuni>
you can't do much
<antocuni>
tp_traverse does not receive a ctx, so you cannot call any API function
<cfbolz>
right
<cfbolz>
good way to stop weirdness ;-)
<arigato>
tp_traverse should end up being called only once on an object when the GC determines that it's alive, never several times
<cfbolz>
arigato: ok, but the referenced objects can already have been moved
<arigato>
yes, same as with any reference anywhere
<cfbolz>
but yes, without a ctx you cannot call any hpy api functions on them at all
<arigato>
ah yes, indeed
<arigato>
tp_traverse should not try to do anything with the handles, of course
<cfbolz>
arigato: it being C there's little we can do to stop people. except not give them a ctx ;-)
<antocuni>
IIRC we had discussions to introduce a "get_global_ctx()" function in HPy because it was useful/needed for some use case, but I don't remember what were the outcome and whether it has been introduced
<antocuni>
but even in that case, we should just document that you should not call it from tp_traverse()
<arigato>
antocuni: maybe objectmodel.hlinvoke()?
<antocuni>
what the hell is that? :)
<arigato>
unsure it's easier to use than doing it all manually, where "all" is:
<arigato>
something like writing a function and instead of @specialize.memo, use something custom class Entry(ExtRegistryEntry)
<arigato>
visit = fetch_visit(callback)
<arigato>
class Entry(ExtRegistryEntry): _about_=fetch_visit
<arigato>
then you have methods in the Entry, like compute_result_annotation and specialize_call
<arigato>
which have annotations and rtypes for the full callback
<antocuni>
this would work also for callbacks-which-are-methods, or only for callbacks-which-are-functions?
<arigato>
so for example, you take R = rtype of the callback, and use that in the generated visit function in hlinvoke(R, callback, voidp_arg)
<arigato>
I may be extended to work with callbacks-with-are-methods, but it's even more mess, so maybe let's not
<antocuni>
I'm unsure to fully understand how it will work but I suppose I can try
<arigato>
I don't know if I'm talking sense or not at all
<antocuni>
"good" :)
<arigato>
maybe you can use rffi.cast(VISITPROC, llhelper(FUNC, callback)), where VISITPROC = function-with-one-voidp-arg, and hope for the best
<antocuni>
because they have similar-enough signatures that it works if the stars are aligned?
<arigato>
yes
<antocuni>
it's a bit worrying but it's worth trying I suppose
<antocuni>
arigato: do you have any C source of pypy around on bencher7, just to check what is the C signature of function-callbacks?
<arigato>
look in ~/pypysrc/*compiled/ but I don't remember on bencher7 specifically
<antocuni>
nope, there is no *compiled
<antocuni>
ok, I'll just do a -O2 translation, they are fast nowadays
pmp-p has quit [Ping timeout: 240 seconds]
pmp-p has joined #hpy
pmp-p has quit [Quit: No Ping reply in 180 seconds.]
pmp-p has joined #hpy
DuToit has quit [Ping timeout: 276 seconds]
DuToit has joined #hpy
DuToit has quit [Ping timeout: 244 seconds]
FFY00 has quit [Read error: Connection reset by peer]
FFY00 has joined #hpy
DuToit has joined #hpy
DuToit has quit [Ping timeout: 244 seconds]
DuToit has joined #hpy
DuToit has quit [Read error: Connection reset by peer]
DuToit has joined #hpy
DuToit has quit [Ping timeout: 260 seconds]
DuToit has joined #hpy
FFY00 has quit [Read error: Connection reset by peer]