<cfbolz>
Still, that takes like 20min on my oldish laptop
<fijal>
or at least a rubber duck ;-)
<fijal>
cfbolz: there is an extra problem which is now we need to make our tests pass under pypy
<fijal>
probably a good thing to fix, but we have been dreading that
<cfbolz>
fijal: I have been dogfooding on my laptop
<cfbolz>
The situation is rather reasonable
<fijal>
cool!
<fijal>
I'm still kinda fighting with bootstrapping issues
<fijal>
the system libffi is broken (likely at least), so I can't use the python I got
Guest96 has joined #pypy
<fijal>
compiling new python is not really a supported thing?
<fijal>
so I'm compiling pypy, but it crashes
<fijal>
cfbolz: do you have any clue as to why we need to call write_protect in the GC? what's executable in the GC for custom trace hook?
<fijal>
ah I suppose MAP_JIT should not always be used is the answer
<mattip>
fwiw, the windows buildbot uses pypy2 as a runner everywhere, it seems to have very few rough corners
<arigato>
fijal: you're having troubles with a no-jit pypy?
<arigato>
yes, MAP_JIT should never be used in a no-jit pypy, and only carefully used in a jit-pypy
<arigato>
is there nowadays a proper documentation of what MAP_JIT does? for the longest time there was only vague words
<fijal>
I don't think MAP_JIT is being called with no jit pypy
<fijal>
I mean, it is, because it needs to be called for cffi
<fijal>
I think
<fijal>
but I'm trying to understand why in those patches submitted there is a write_protect call in trace_drag_out, that seems wrong
<arigato>
indeed
<fijal>
as far as I understand, what you do:
<fijal>
1. you create mmap with PROT_EXEC | PROT_WRITE
<fijal>
2. you call write_protect(0); write; write_protect(1) each time you want to write there (we probably need to move "copy to the final destination" somewhere in the JIT, otherwise it would be too nested
<fijal>
3. you can execute that code at any other time
<fijal>
MAP_JIT goes to 1.
<fijal>
I think I need to go take a walk for a second
<arigato>
yes, that seems right to me
<fijal>
ok, then why does it crash is probably an extra question
<fijal>
from what I can tell, the MAP_JIT is really not called from the GC
<arigato>
in the issue I argued for moving the write_protect() calls around a larger portion of the jit backend, but yes, it would work too if we put it really just around the actual writes to MAP_JIT memory
<fijal>
but I might be wrong, maybe there are custom handlers for callbacks? but no, GC callbacks should be prebuilt, no?
<arigato>
they should, yes
<fijal>
right, are we sure that none of the "larger JIT backend" can call anything that will make write_protect calls nested?
<fijal>
that's a bit my worry
<arigato>
ah, no
<arigato>
makes sense then
<fijal>
obviously if we call write_protect in the GC, then it's bad
<fijal>
but I don't think we should? why do we do this?
<arigato>
we shouldn't
<arigato>
so we should add some write_protect calls that are *all* around *just* the place that writes to MAP_JIT memory
<arigato>
with some try:finally: to be on the safe side
<fijal>
yes, can you look at the things I added so far? I think they're all pushed (they're not in try: finally: though)
<fijal>
I think most of those places are VERY BAD if exception happens though
<arigato>
likely
<fijal>
pushed last one
<arigato>
ah, you need to call write_protect() around calling the libffi's ffi_prep_closure_loc? OK
<arigato>
I bet some future release of libffi-for-m1 will call write_protect() itself, but that should still work
<fijal>
I think so? maybe there is a more clever way of calling it so both regions are not the same
<arigato>
right, but don't do it, because that solution is the one with a gaping security hole that was never addressed
<cfbolz>
fijal: I think we can have a tiny bit larger scopes for the write_protect(0) calls, just around the backend maybe. but that's all for later, just an optimization
<fijal>
arigato: compiling pypy in 5 min is crazy though (and I don't have a JIT on me yet)
<arigato>
wow
<arigato>
seems that Intel is being left in the dust
<fijal>
I mean, it's not TOO surprising that if software people talk to hardware people, good things happen
<cfbolz>
yes
<cfbolz>
and with our experience with intel, it seems they have a hard time talking to intel people
<arigato>
reading rpython/rlib/clibffi.py, maybe we should stop using the allocated memory to store a linked free-list, because it doesn't make sense to have to call write_protect() to update that free-list
<cfbolz>
*to software people
* cfbolz
needs more coffee
<arigato>
:-)
<arigato>
in skip_if_m1(), is that because CPython's libffi is buggy? in that case, should we skip only on CPython and not on PyPy2?
<arigato>
<end of my comments>
<fijal>
arigato: yes, I agree
<fijal>
arigato: but I want pypy to run any tests first, before I dig into that :-)
<fijal>
we should use, I think, the allocation method in libffi itself and not our own freelist
<fijal>
(then presumably they would call write_protect for us)
<arigato>
I'm wary of that
<fijal>
the current crash is "illegal instruction" through some cffi stuff
<fijal>
arigato: so I should just use a different allocation mechanism for free list and that's it?
<arigato>
I think the allocation method in libffi itself is tailored for the two-addresses-one-for-writing-one-for-executing, and that's what I *think* we absolutely don't want
<arigato>
yes
<fijal>
ok
<arigato>
of course I may be wrong, there was an experiment a couple of years ago in libffi on one platform that was really doing the right thing. I have no idea if the m1 port is done the same way
<fijal>
break
<arigato>
and maaaaybe that would mean that you have to use the allocation method in libffi itself, and the existing libffi API still works but only if you do that
<arigato>
but that's only a guess
<fijal>
I'll test it out, I think
<arigato>
that seems like a good idea
<fijal>
it's funny, I think the development style would not work any other time, but 5min to compile pypy means you can really just retranslate if you feel like it (so no real reason to fight with broken libffi)
<fijal>
I think I want to also test in C if libffi is working at all, would be a good step one
<arigato>
agreed
<fijal>
qsort crashing is a bit bad, I think
<arigato>
uh?
<fijal>
arigato: I can't get qsort example from tests to pass in C without using ffi_closure_alloc
Julian has joined #pypy
<arigato>
:-/
<fijal>
well, maybe we should use the official interface then
Julian has quit [Quit: leaving]
<cfbolz>
fijal: hehe, the fact that I use pypy locally for tests means that I commit tests that then break on cpython on the buildbot ;-)
<fijal>
haha
<fijal>
ok, I think I got somewhere using ffi_closure_alloc, maybe
<fijal>
cfbolz, arigato: right, there is something in the way that's quite annoying - we can't do tests with memory_protect at all, because it probably interferes with the underlaying libffi doing those calls?
<fijal>
I have two layers of libffi calls, I suppose calling libffi via libffi full on does not work with calls around it
<arigato>
hum, yes, that looks likely
<arigato>
maybe avoid libffi completely to call memory_protect?
<arigato>
e.g. use cffi in API mode
<arigato>
hum, unclear that it would help, actually
<arigato>
on pypy, even in API mode, the functions are first called with libffi, I think, and only the JIT compiler turns that into a real call
lritter has joined #pypy
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Guest96 has joined #pypy
Guest96 has quit [Ping timeout: 256 seconds]
<fijal>
it's not the memory_protect call
<fijal>
time make lldebug0
<fijal>
18s
<fijal>
mattip: how do I invoke _ssl_build.py by hand? it needs some flags for sure
<fijal>
ok, I think I found new problem, the libffi linked is one from /usr/lib, not the one from homebrew (and I'm sure that's wrong?)
<fijal>
am I sure actually?
<fijal>
wat
<fijal>
OS X is very very strange....
<fijal>
I think I gonna call it for the day though
<mattip>
cd lib_pypy; pypy3-c pypy_tools/build_cffi_imports.py --only _ssl --embed
<mattip>
will download and build openssl and then statically link _ssl.*.dylib with it
<mattip>
cd lib_pypy; pypy3-c _ssl_build.py
<mattip>
will build _ssl and dynamically link with whatever openssl it can find
<fijal>
I need more magic incantations I think with some flags
<mattip>
you can add them in lib_pypy/_cffi_ssl/_cffi_src/build_openssl.py, see _extra_compile_args