cfbolz changed the topic of #pypy to: #pypy PyPy, the flexible snake | IRC logs: and | Matti: I made a bit of progress, the tests now only segfault towards the end
f4at has joined #pypy
f4at has quit [Quit: bye]
Atque has joined #pypy
Atque has quit [Remote host closed the connection]
Atque has joined #pypy
Atque has quit [Ping timeout: 240 seconds]
<arigato> in link with issue #2889, maybe I should mention that we could still try to embark on the larger project to change the way tuples are rtyped, from GcStruct to a new XxxStruct
<arigato> which would be exploded in its individual fields in C functions, and with new argument and return value passing conventions, and the largest part is to handle storing it in other data structures (then it becomes a regular non-GC Struct I guess)
<arigato> the goal being to stop once and for all to have these issues that occur because a function is returning a tuple
<arigato> or a tuple is passed to a residual call and this makes the GC pressure increase
Atque has joined #pypy
<arigato> (note that "exploded in C functions" is what I would have said 10 years ago, but maybe nowadays the C compilers are all good enough to have the same result)
Guest96 has joined #pypy
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Guest96 has joined #pypy
Atque has quit [Ping timeout: 240 seconds]
Atque has joined #pypy
otisolsen70 has joined #pypy
<cfbolz> arigato: I wonder whether we really need a new type
<cfbolz> Or whether it should be done on a best effort basis in the backend opts
<cfbolz> So we would 'just' need the concept of a return pointer argument
<arigato> right
<arigato> maybe expressed as a regular Ptr(Struct), and we change the signature of functions to pass the pointer to the struct that we want to be filled; this would require only a single new llop, "alloca", which can be implemented without actually calling alloca() but same idea
<cfbolz> We used to have an alloca 😅
<arigato> yes
<cfbolz> arigato: does not sound too terrible. I'm unsure about the effects on the GC transformer
<cfbolz> And don't quite know how it would work in the JIT
<arigato> why the GC transformer?
<cfbolz> arigato: does it affect the roots?
<arigato> yes, the JIT is some work too. maybe that would be a good reason for exploding the fields into local variables, to make the JIT's work easier
<arigato> and ah, yes, the roots
<arigato> another reason to explode the fields, then the roots are regularly handled
<cfbolz> Right
<fijal> hm
<cfbolz> arigato: so you would basically only have a type that is conceptually a box that you can store a return value into
<cfbolz> Which turns into a pointer to a local variable in C calls
<arigato> the C call itself would be "allocate the box, call the function, read the fields out of the box, and forget the box immediately", so that we're sure there are no GC roots lurking in there?
<cfbolz> Yes. Using alloca?
<arigato> in C, just using a local variable of type struct
<cfbolz> Ah of course
<cfbolz> But what about the JIT? Where does it make space? The CPU stack explicitly?
<arigato> yes
<arigato> meh
<cfbolz> Needs a bit of care
<arigato> it would be easier if we had just a lloperation that does it all including the call
<fijal> llop.call_with_multiple_return_values?
<arigato> maybe we can keep the existing FuncType declared as Void return and with an extra first argument that is a pointer to the resulting struct,
<arigato> but indeed have an optional llop.call_with_multiple_return_values() when we call this kind of FuncType
<cfbolz> It does not fit the flow graph model so well
<arigato> indeed
<arigato> vbox = llop.call_with_multiple_return_values(func, arg1, arg2...)
<arigato> res1 = llop.extract_return_from_box(vbox, 0)
<arigato> etc.
<arigato> ?
<cfbolz> That's super close to a structs :-)
<cfbolz> But yes
<arigato> the jit code writer can check that call_with_multiple_etc is immediately followed by extract_return_from_box, and consolidate it all---I think the jit opcodes can have multiple return values?
<arigato> maybe not?
<cfbolz> arigato: no, they can't I think
<cfbolz> It's a bit of a mess, and the question is a bit where we move that mess
<arigato> yeah
<cfbolz> It would add a bit of an implicit structure to how operations around a call must appear
<arigato> yes, not good
<cfbolz> We have a few elements of that already in some places already
<cfbolz> Eg the guard_no_exception after a call in the JIT
<arigato> a differently obscure idea:
<arigato> v0 = result_of_later_call(0)
<arigato> v1 = result_of_later_call(1)
<arigato> call_with_multiple_results(func, v0, v1, arg0, arg1, arg2...)
<arigato> in C it's translated as func(&v0, &v1, arg0, arg1, arg2...)
<arigato> the JIT would produce a residual call func(&v0, &v1, arg0, arg1, arg2...) where v0 and v1 are in the jitframe
<arigato> so the call_with_multiple_result "spills" v0 and v1 in the sense that after the call, the values are living in the jitframe, and can be reloaded from there if needed later
<cfbolz> arigato: right. Still a dependency
<cfbolz> But at least not operations before *and* after the call
<arigato> only a weak one. It breaks the idea of immutability. the result_of_later_call() are just placeholders that can occur earlier too, as long as you don't use v0 and v1 before the actual call is made
<cfbolz> Yes
<arigato> (I think it doesn't need to have any argument, the index 0 or 1 is pointless)
<cfbolz> And the type of v* is just the lltype
<cfbolz> Nothing special
<arigato> yes
<cfbolz> It can even be chained in theory
<cfbolz> If the result comes from a recursive call
<arigato> the gc transformer "should" not worry about gc roots between the result_of_later_call and the actual call_with_multiple_xxx
<arigato> but result_of_later_call can initialize the variable with null to be on the safe side
<cfbolz> Yes, was just about to say
<arigato> chaining: no, I have in the mind that the FuncType should start with types like lltype.Ptr(lltype.Signed)
<arigato> I think it needs to be lltype.Ptr(lltype.Struct(lltype.Signed)) or something like that
<arigato> then the return from that function writes each result to its independent location inside a struct-of-one-field
<cfbolz> It's an array of length 1 I suspect
<arigato> ah yes
<cfbolz> Ok I can see how this works with primitive types
<cfbolz> Does it also work for the GC references?
<arigato> it should
<cfbolz> Who keeps them alive?
<arigato> er, maybe?
<arigato> tempted to answer "nobody but it doesn't matter"
<cfbolz> But then we have a dependency again
<cfbolz> You really have to 'write and immediately return'
<cfbolz> In the called function
<arigato> ...yes
<cfbolz> Hmmm, 'clearly': we should not use a pointer to a local variable in C, but a pointer into the shadow stack
<arigato> oh
<cfbolz> In the JIT, that's what the plan with the JIT frame already does
<cfbolz> Really obscure though
<arigato> the shadowstack never moves around, right?
<cfbolz> No
<arigato> then maaaybe?
<cfbolz> :-)
<cfbolz> We should put this elaborate plan into an issue and then simply not implement it
<arigato> fwiw, we already have an operation 'gc_restore_root' which takes an argument that in C is actually an output argument
<arigato> this llop occurs just after a call, after the graph is gc-transformed
<arigato> so the right thing should occur if we just genc the operation call_with_multiple(func, gcres0, ..) by passing the address in the shadowstack
<arigato> because it's re-read from there immediately afterwards
<arigato> OK, it's a plan that is elaborate and won't get implemented
jacob22 has quit [Ping timeout: 248 seconds]
Guest96_ has joined #pypy
Guest96 has quit [Ping timeout: 244 seconds]
Atque has quit [Quit: ...]
<fijal> haha
<mattip> we have lost our macOS buildbot, which was supported by the CPython buildbot worker owner,
<mattip> he was having problems with python2 last I heard
* fijal struggles to parse cpython buildbot worker owner
<mattip> the person behind the macOS buildbot for CPython
Atque has joined #pypy
<mattip> I guess I can try to run the buildbot worker on my M1 machine in x86_64 mode
otisolsen70_ has joined #pypy
otisolsen70 has quit [Ping timeout: 240 seconds]
otisolsen70__ has joined #pypy
<fijal> this would take a while....
<fijal> mattip: did we not have an offer to use some mac somewhere?
otisolsen70_ has quit [Ping timeout: 244 seconds]
Atque has quit [Quit: ...]
otisolsen70__ has quit [Quit: Leaving]
Atque has joined #pypy
Guest96_ has quit [Quit: Textual IRC Client:]
jacob22 has joined #pypy
lritter has joined #pypy
reneeontheweb has joined #pypy
<cfbolz> fijal: btw, the biggest class of tests that seems to be a problem for pypy is boehm-using stuff I think
reneeontheweb49 has joined #pypy
reneeontheweb has quit [Ping timeout: 252 seconds]
<mattip> fijal: I would hope that if there was such an offer I would have encouraged them to set up a second worker
[Arfrever] has quit [Killed (NickServ (GHOST command used by [Arfreve1]))]
[Arfrever] has joined #pypy
otisolsen70 has joined #pypy
otisolsen70 has quit [Ping timeout: 246 seconds]
jacob22 has quit [Ping timeout: 248 seconds]
Atque has quit [Quit: ...]
<fijal> mattip: ok, maybe we should ask?
<fijal> cfbolz: right
<fijal> jit boehm or just boehm? and why?
Dejan has joined #pypy
reneeontheweb49 has quit [Quit: Client closed]
<cfbolz> fijal: I haven't looked at all, honestly
<cfbolz> I think it's JIT tests and C backend tests
<antocuni> I'm in gdb and I have a pointer to W_Root:
<antocuni> l_w_target_6 = (struct pypy_pypy_interpreter_baseobjspace_W_Root0 *)l_obj_20538; │
<antocuni> $23 = {wr_super = {_gcheader = {h_tid = 140737269518192}}}
<antocuni> how can I retrieve its actual RPython type?
<antocuni> I guess it's stored somehow in the h_tid
<cfbolz> antocuni: there's a script in pypy/tool I think
<cfbolz> it has a comment that explains it
<cfbolz> I can never remember, but that's where I always look
<antocuni> ah, there is which contains a command rpy_type
<antocuni> I think there is a good chance that I am the one who wrote it 😂
<antocuni> it seems it's no longer working though
<antocuni> and I'm having a hard time finding the source code which explains what's inside h_tid
<cfbolz> antocuni: that's a mess
<cfbolz> it's an offset into a collection of all the types
<antocuni> yes, that's the easy part
<antocuni> the hard part is to understand which bits encode the offset
<cfbolz> half of them :-)
<cfbolz> I can find out, sec
<antocuni> according to gdb_pypy, I should do "offset = tid & 0xFFFFFFFF"
<cfbolz> antocuni: how does the gdb script fail?
<cfbolz> it worked recently for me
<antocuni> but this gives an offset of 0xf2f4cf70
<antocuni> and typeids.txt contains only 4688 lines
<cfbolz> antocuni: no, it's not an *index*
<cfbolz> it's an offset in bytes into the vtables
<antocuni> ah
<cfbolz> but the number still sounds high
<arigato> antocuni: that's bogus
<arigato> the value stored in tid is '0x7ffff2f4cf70'
<arigato> which is a pointer
<arigato> i.e. it's not a tid at all
<antocuni> ah, so something is completely wrong
<arigato> yes
<cfbolz> antocuni: there are no tid in the object headers any more
<cfbolz> only briefly looking at it, the script has the complicated parts I think? maybe something shallow to fix?
<antocuni> I'm lost
<antocuni> so, I don't understand whether my assumptions is wrong and h_tid is SUPPOSED to be a pointer to something
<antocuni> or my memory is corrupted and h_tid contains a bogus value
<cfbolz> antocuni: for me the script works
<antocuni> for me it works for some values, but not for something else
<cfbolz> the tid you posted above looks maybe corrupted
<cfbolz> it's neither an index nor a pointer
<cfbolz> the lowest four bytes are 4076130160
<cfbolz> that's an offset of several GB
<antocuni> pypy_g_list_416 is the list of hpy handles
<antocuni> and if you print it, it seems to contains reasonable stuff, which is pretty printed because I loaded gdb_pypy
<antocuni> so e.g. rpy_type works on the list itself and on l_items
<cfbolz> so it sounds to me that you potentially have memory corruption
<antocuni> yes, maybe
<antocuni> let's see what happens with a fresh process
<arigato> or gdb is confused and that value for the local variable 'obj' at this point in time is bogus
<antocuni> I think I'm even more confused than gdb at this point :)
<antocuni> note that memory corruption is possible and even likely, since I'm playing with a custom GC tracer for HPy objects
<antocuni> so it's very possible that I screw everything
<antocuni> is there an easy way to retrieve e.g. space.w_None from gdb?
<cfbolz> antocuni: it's maybe just "pypy_g_pypy_objspace_std_noneobject_W_NoneObject"
<antocuni> right
<antocuni> ok, rpy_type is definitely bogus
<antocuni> because it fails in the same way even if I put a breakpoint in main()
<cfbolz> antocuni: 'bogus' as in the gdb script is broken?
<antocuni> aaargh
<antocuni> I'm stupid
<antocuni> the problem is that I was missing a level of indirection
<antocuni> to access the items of pypy_g_list_416 you need to do:
<antocuni> (gdb) p pypy_g_list_416.l_items->items[2]
<antocuni> instead, I was doing pypy_g_list_416.l_items[2], which goes to random memory
<antocuni> so, if you pass an actual object, rpy_type works well
jacob22 has joined #pypy
<cfbolz> antocuni: eh
<cfbolz> antocuni: 'good'
jacob22 has quit [Ping timeout: 272 seconds]
Guest96_ has joined #pypy
Guest96_ has quit [Client Quit]
jacob22 has joined #pypy
tos9_ has joined #pypy
lritter has quit [Quit: Leaving]
tos9_ is now known as Julian
Dejan has quit [Quit: Leaving]
Julian has quit [Quit: leaving]