<antocuni>
what the hell is lltypesystem.llgroup.GroupType?
<cfbolz>
antocuni: that's what you always get confused about ;-)
<cfbolz>
antocuni: it's the container for all vtables
<nimaje>
reads to me like some way to save memory by storing all possible objects in a static array and use an index into that array as reference (and using some type that is smaller than a pointer to really save memory)
<cfbolz>
nimaje: exactly
<antocuni>
ah, didn't know about this optimizatio
<antocuni>
which is getting into my way, obviously
<cfbolz>
antocuni: it's how we store both a type pointer *and* gc flags in one word
<antocuni>
because the type pointer is not really a pointer?
<cfbolz>
indeed
<antocuni>
ok, got it
<antocuni>
I stumbled upon it because for some reason I want to call ll2ctypes on it
<antocuni>
but I haven't understood exactly what's happening yet
<antocuni>
this needs to be run on top of ac53ab912657, which contains my other branch gc-custom-trace-memo, in which all GC callbacks have been turned into functions (as opposed to bound methods)
<antocuni>
with the idea that we can finally @specialize.memo() on it
<antocuni>
I think that this is caused by trying to @specialize.memo() on a callback which takes "gc" as an argument. It might be related to the fact that "gc" has a special type because GCBase._alloc_flavor_ == 'raw'
<antocuni>
but I don't really know
<antocuni>
something seems to be very wrong: basically we are hitting rpbc.py:convert_desc at line 280, where there is a comment saying:
<antocuni>
"extremely rare case, shown only sometimes by test_bug_callfamily: don't emit
<antocuni>
be called anyway.
<antocuni>
NULL, because that would be interpreted as equal to None... It should never
<antocuni>
"
<antocuni>
and then it calls the rffi.cast which ultimately leads to the crash. But I think that the problem is that probably we should NOT hit the "extremely rare case"
<antocuni>
but I'm a bit lost I've never really understood what are functions descs :)
<cfbolz>
antocuni: I think nobody is left who does
<antocuni>
that's what I feared
<arigato>
antocuni: possibly, it means that the rtyper didn't completely analyze your test's rpython functions
<arigato>
something like calling gc.trace(callback) but the callback is not rtyped for some reason
<arigato>
then the line "gc.trace(callback)" would end up in this line 280: trying to turn 'callback' from a constant to a function pointer, but we don't have a function to point to
<arigato>
'test_bug_callfamily' gives an example where the function cb1 is never annotated at all because it's not called. It is still passed as an argument of g(), but g() never calls it. It must still be turned into a function pointer because other places call g() with cb2, which is a real function that is also called somewhere else
<arigato>
so maybe you're getting something similar if gc.trace(callback) is missing the logic that would annotate 'callback' as a potential call
jacob22 has joined #pypy
<antocuni>
arigato: yes, I imagined that it was something like that
<antocuni>
but it's weird because this is not a test, it's the real translation
greedom has quit [Remote host closed the connection]
<antocuni>
so it's weird that GC callbacks are not annotated. Maybe the problem is that these specific GC callbacks will be annotated later?
greedom has joined #pypy
<arigato>
dig around in pdb++, the above is just a guess from limited information
<antocuni>
arigato: I'm not sure where to look, but e.g. this seems related
<antocuni>
Pdb+ on Pdb++) self.uniquerows
<antocuni>
[{<FunctionDesc for <function _append_rpy_referent at 0x0000000002abadb8>>: <* fn _append_rpy_referent>}]
<antocuni>
if I understand correctly, it means that _append_rpy_referent is the only function which was actually seen by the rtyper for the callbacks
<arigato>
you should go "up" a few times and figure out which operation it is rtyping, too
<arigato>
check if it's really the direct_call to gc.trace()
<arigato>
maybe you'll find out that it's a different call to gc.trace() which tries to pass a different callback than _append_rpy_referent
<antocuni>
it's not a call to gc.trae
<antocuni>
the graph is: <FunctionGraph of (pypy.module._hpy_universal.apiset:147)ctx__trace_one_field__star_2 at 0x2a15360>
<antocuni>
and the op is: v32 = simple_call((function _trace_one_field), (function gc_callback__ref), v30, v31)
<arigato>
ah OK
<arigato>
you should have the concrete Python function object, too, the one it tries to convert to a desc
<arigato>
("down" a few times)
<antocuni>
yes, it's <function gc_callback__ref at 0x000000000468b808>
<antocuni>
arigato: I'm doing this inside a screen on bencher7, if you want to see it
<arigato>
and the function _trace_one_field doesn't end up calling its first argument?
<arigato>
(here gc_callback__ref?)
<antocuni>
ah
<antocuni>
indeed, it doesn't call it yet
<antocuni>
I "just" wanted to see whether it translated before going further
<arigato>
OK :-)
<antocuni>
so, you think that inserting a call to gc._trace_callback(callback, ...) should be enough?
<arigato>
probably
<antocuni>
ok, I'll try
<arigato>
probably not worth trying to figure out an even more obscure hack to fix the exact crash
<antocuni>
although calling gc._trace is it's own (different) mess, because I now need to figure out how to pass gc, arg1 and arg2 to the ll function
<antocuni>
my plan is to stick them to some global object
<antocuni>
"work" in the sense that it translates, and it seems to have a slim chance to be vaguely correct
<arigato>
well, my guess is that this works even though it doesn't call 'callback', because there is only one such function, not several ones like in the real pypy
<antocuni>
arigato: no, it contains all the callbacks because it's a translation target, so it is translated with the full GC
<antocuni>
and it DOES call the callback, in gc._trace_callback at line 56
<antocuni>
and it "almost" translates also in full pypy
<antocuni>
"almost" because I hit the next yak: if I call "storage.tp_traverse()" (which is a C function pointer of type HPyFunc_traverseproc), rpython complains that:
<antocuni>
the custom trace hook <function hpy_customtrace at 0x0000000004ed3718> for <GcStruct HPyStorage { tp_traverse, data }> can cause the GC to be called!
<antocuni>
I suppose the fix is to somehow convince rpython that this specific call cannot allocate memory, although I'm not sure how
<cfbolz>
antocuni: I think there is a way to mark llexternals as 'I'm actually harmless'
<antocuni>
do you know how?
<antocuni>
I'm looking at CollectAnalyzer and I don't see any relevant flag for llexternals (only for direct_call)
<arigato>
llexternal() has a flag random_effects_on_gcobjs=False
<antocuni>
right, but tp_traverse is not an llexternal
<antocuni>
it's a function pointer which we have got from somewhere else
<antocuni>
like, provided at runtime by the user
<antocuni>
probably the easiest way to add a flag to hpy_customtrace to skip the analysis completely
<arigato>
maybe try to make a lltype.functionptr() with the same flag as llexternal() puts when you say "random_effects_on_gcobjs=False"?
<arigato>
or that
<antocuni>
I'm confused, what is an lltype.functionptr()?
<arigato>
you can either read through lltype, or ignore and try to skip the analysis completely
<antocuni>
I think I'll just skip the analysis :)
derpydoo has joined #pypy
<antocuni>
hoorray, I got something which seem to work 🎉
<arigato>
:-)
<antocuni>
still not complete because it always visit only the first field, but that one field is actually reported by gc.get_referents()
<antocuni>
and now everythig works 🎉🎉🎉
<antocuni>
this was probably one of the features with most yaks to shave which I ever done in pypy :)
greedom has quit [Remote host closed the connection]
<LarstiQ>
antocuni: so, blogpost next? ;)
alice has quit [Read error: Connection reset by peer]