cfbolz changed the topic of #pypy to: #pypy PyPy, the flexible snake | IRC logs: and | hacking on TLS is fun, way more fun than arguing over petty shit, turns out
stkrdknmibalz has joined #pypy
the_rat has quit [*.net *.split]
ronan has quit [*.net *.split]
ronan has joined #pypy
the_rat has joined #pypy
Atque has joined #pypy
greedom has joined #pypy
greedom has quit [Remote host closed the connection]
greedom has joined #pypy
greedom has quit [Remote host closed the connection]
<cfbolz> cool, now the interpreter/ tests all pass
<cfbolz> with the new parser
<cfbolz> now testing whether we can parse the whole stdlib
<ctismer> cfbolz: I have verified that builtin methods have no __module__ attribute, others have.
<cfbolz> cool
<cfbolz> stdlib now parses 🎉
stkrdknmibalz has quit [Quit: WeeChat 3.0.1]
<ctismer> cfbolz: I think it might be a good idea to redefine `PyMethod_Check` for CPython users:
<cfbolz> ctismer: not a bad plan!
<cfbolz> mattip: what do you think?
<ctismer> cfbolz: My tests went from 99 bugs down to 67 :)
<ctismer> 76
<cfbolz> nice!
<LarstiQ> oh no, they're going back up again! ;)
<mattip> isn't the fact a method is a builtin an implementation detail?
<mattip> I would like to understand more why this is needed
<cfbolz> mattip: it totally is, but a lot of people seem to depend on it
<ctismer> mattip: In CPython, you have this PyCFunction structure which shows up as "builtin function or method". In Python, you have "method" instead, which means it is written in application level, so to speak. For CPython users, this difference is very visible and missing in PyPy.
<ctismer> mattip: Although it is an implementation detail, the answer of PyPy is totally unexpected for Python users. Isn't that the audience you want to please? :)
<mattip> what do you do differently if you detect one or the other?
<mattip> what happens if you get the "wrong" answer?
Atque has quit [Quit: ...]
<ctismer> mattip: see yesterdays cpne example `BindingManager::getOverride`. That override function wants to find out whether a builtin method is over-written by Python. Then it is handled very differently, needs to hold the GIL, etc.
<ctismer> mattip: Sure, we could also build a registry for internally known methods, but this was never a problem in the pre-PyPythonic time.
<ctismer> mattip: by this fix, we got 23 errors less
<mattip> I still don't understand why the tests are failing.
<mattip> On the one hand you say you don't want a registry, but it seems 23 tests do check that specific functions behave in a certain way
<mattip> so on PyPy you hold the GIL, etc in different places than on CPython, what is the damage that is caused?
<mattip> maybe PyPy's implementation of some builtin requires the GIL, is releasing the GIL a defined behaviour for builtins?
<mattip> maybe we should propose to CPython there be a flag on methods if they require the GIL or not
<mattip> but again, I don't understand so please help me see why this all is needed
charlie_sando has joined #pypy
<charlie_sando> Well I came to ask a bout installing pandas for the latest pypy but it just finished :D
<charlie_sando> I'm getting this error when trying to install psycopg2, "ImportError: /home/q/anaconda3/envs/pypy/lib/pypy3.8/site-packages/psycopg2/ undefined symbol: PyInterpreterState_Main
<charlie_sando> " , any ideas on how to resolve ?
<ctismer> mattip: no, totally misunderstood. There are different reasons, the GIL is only part of the story. There are functions closely implemented to Qt needs, there is special handling of Qt needs and quirks. That is handled in C++ helper functions which are explicitly not meant as user code. They follow special rules. The GIL is such a rule. These function are called very frequently, and if these would acquire the GIL, the whole GUI would
<ctismer> freeze because of the so-called `GIL-Carousel`. Every Enter-Exit of that costs 5ms -> freeze
<mattip> charlie_sando: use psycopg-cffi
<ctismer> mattip: With my hack, this part works exactly as before. We don't need any patch tho, we are fine with our `PyMethod_Check` redefinition.
<charlie_sando> mattip, worked ty!
<ctismer> mattip: Having PyPy functions pretending to do the same as CPython's, but doing something differently has cost us very much time. Code that used to work suddely stops to work because PyPy has a different understanding of that. It would have been much better for us when PyPy would rename their only 70% COM
<ctismer> ..compatible functions into something slightly different (like PyPyMethod_Check), because then we would not run into cases which you only can understand if you always work with the compatibility list.
<mattip> ctismer: the documentation of PyMethod_Check is quite explicit
<mattip> does PyPy not comply with the spec?
wleslie has quit [Quit: ~~~ Crash in JIT!]
<mattip> I am glad you found a way to get what you wanted
<mattip> it seems you want a IsBuiltin
<mattip> or IsOKToCallWithoutTheGil()
<mattip> although looking at some of the builtins
<ctismer> mattip: The function is completely correct. But PyCFunction is not a method in CPython, but _is_ suddenly in PyPy. Therefore I would either re-build the CPython behavior or rename the function, because it has different scope.
<ctismer> mattip: Sure this is a weird CPython quirk, agreed.
<mattip> is it always safe to call a PyCFunction without the GIL?
<ctismer> mattip: That depends on the function. It is for us, because we know what they are (not) doing.
<ctismer> mattip: But everything that might call back into the interpreter must hold the GIL.
<ctismer> mattip: Even a Py_DECREF ðŸĪŠ
<mattip> cool. But now I am confused how this relates to PyMethod_Check and PyCFunction
<mattip> you have a set of functions you know you can treat specially, and you identify them by calling PyMethod_Check ?
<mattip> which is a generic test for isinstance(types.MethodType)
<ctismer> mattip: No, vice versa. All the functions where we need to be careful are those which are written in Python. The functions where we have full control are the C functions, which are PyCFunction objects.
<ctismer> They are the default functions when there is no override. When someone declares a class with an override method, then this comes from Python and needs the GIL. But that are very few, so we can neglect the delay.
<ctismer> mattip: For instance, `QApplication` has hundreds of methods, one is `notify`. This notify is used heavily in the GUI and written in C. Somemone derives a class from `QApplication` in Python with his own `notify` function. That is written in Python and treated differently.
dmalcolm_ has joined #pypy
<ctismer> mattip: Our problem was only that PyMethod_Check is extended in PyPy to include the cases which are PyCFunction in CPython. That is fixed by now. Of course, a registry would be cleaner, hint taken 😆
dmalcolm has quit [Ping timeout: 260 seconds]
<mattip> ahh, so when calling notify from C, you want to know if it is your notify or one from a user, got it
<ctismer> mattip: 🖖
<mattip> could a user override notify via C code with a version that calls back to python, and then it is a PyCFunction, no?
<mattip> and by chance 98% of users do this in single-threaded code, so it is not critical whether the GIL is held or not
<mattip> (or at least code where notify and whatever it does is not shared across threads)
<ctismer> mattip: sure he could. But then he does C code, playing the same game as we do, and he must use the API rules (which are written nowhere, I guess)
<ctismer> mattip: PySide is written for Python users, oc C programmers who know what they are doing.
Julian has joined #pypy
Julian has quit [Ping timeout: 264 seconds]
Julian has joined #pypy
Julian has quit [Client Quit]
<Hodgestar> ctismer: It sounds like this is a property of the methods one would ideally mark explicitly somehow.
<ctismer> Hodgestar: Yes. But I'm trying to keep the existing (huge) codebase as it is before doing such rigorous changes, just because PyPy and CPython disagree on something :)
Julian has joined #pypy
stkrdknmibalz has joined #pypy
<LarstiQ> very much feels like a Hyrum's Law situation
greedom has joined #pypy
pjenvey1 has joined #pypy
stkrdknmibalz has quit [*.net *.split]
pjenvey has quit [*.net *.split]
Ninpo has quit [*.net *.split]
Julian has quit [Ping timeout: 246 seconds]
Julian has joined #pypy
Ninpo has joined #pypy
Julian has quit [Quit: leaving]
greedom has quit [Remote host closed the connection]
Atque has joined #pypy
pjenvey1 is now known as pjenvey