09:53 <cfbolz> cool, now the interpreter/ tests all pass

09:53 <cfbolz> with the new parser

09:53 <cfbolz> now testing whether we can parse the whole stdlib

10:00 <ctismer> cfbolz: I have verified that builtin methods have no __module__ attribute, others have.

10:03 <cfbolz> cool

10:23 <cfbolz> stdlib now parses 🎉

11:43 <ctismer> cfbolz: I think it might be a good idea to redefine `PyMethod_Check` for CPython users:

11:44 <ctismer> cfbolz https://www.irccloud.com/pastebin/aFa8vwrN/pypy_compat.cpp

11:44 <cfbolz> ctismer: not a bad plan!

11:45 <cfbolz> mattip: what do you think?

11:46 <ctismer> cfbolz: My tests went from 99 bugs down to 67 :)

11:46 <ctismer> 76

11:46 <cfbolz> nice!

12:25 <LarstiQ> oh no, they're going back up again! ;)

12:27 <mattip> isn't the fact a method is a builtin an implementation detail?

12:27 <mattip> I would like to understand more why this is needed

12:30 <cfbolz> mattip: it totally is, but a lot of people seem to depend on it

12:42 <ctismer> mattip: In CPython, you have this PyCFunction structure which shows up as "builtin function or method". In Python, you have "method" instead, which means it is written in application level, so to speak. For CPython users, this difference is very visible and missing in PyPy.

12:44 <ctismer> mattip: Although it is an implementation detail, the answer of PyPy is totally unexpected for Python users. Isn't that the audience you want to please? :)

12:44 <mattip> what do you do differently if you detect one or the other?

12:45 <mattip> what happens if you get the "wrong" answer?

12:47 <ctismer> mattip: see yesterdays cpne example `BindingManager::getOverride`. That override function wants to find out whether a builtin method is over-written by Python. Then it is handled very differently, needs to hold the GIL, etc.

12:49 <ctismer> mattip: Sure, we could also build a registry for internally known methods, but this was never a problem in the pre-PyPythonic time.

12:50 <ctismer> mattip: by this fix, we got 23 errors less

12:51 <mattip> I still don't understand why the tests are failing.

12:51 <mattip> On the one hand you say you don't want a registry, but it seems 23 tests do check that specific functions behave in a certain way

12:52 <mattip> so on PyPy you hold the GIL, etc in different places than on CPython, what is the damage that is caused?

12:54 <mattip> maybe PyPy's implementation of some builtin requires the GIL, is releasing the GIL a defined behaviour for builtins?

12:55 <mattip> maybe we should propose to CPython there be a flag on methods if they require the GIL or not

12:56 <mattip> but again, I don't understand so please help me see why this all is needed

12:58 <charlie_sando> Well I came to ask a bout installing pandas for the latest pypy but it just finished :D

13:10 <charlie_sando> I'm getting this error when trying to install psycopg2, "ImportError: /home/q/anaconda3/envs/pypy/lib/pypy3.8/site-packages/psycopg2/_psycopg.pypy38-pp73-x86_64-linux-gnu.so: undefined symbol: PyInterpreterState_Main

13:10 <charlie_sando> " , any ideas on how to resolve ?

13:12 <ctismer> mattip: no, totally misunderstood. There are different reasons, the GIL is only part of the story. There are functions closely implemented to Qt needs, there is special handling of Qt needs and quirks. That is handled in C++ helper functions which are explicitly not meant as user code. They follow special rules. The GIL is such a rule. These function are called very frequently, and if these would acquire the GIL, the whole GUI would

13:12 <ctismer> freeze because of the so-called `GIL-Carousel`. Every Enter-Exit of that costs 5ms -> freeze

13:12 <mattip> charlie_sando: use psycopg-cffi https://github.com/chtd/psycopg2cffi

13:14 <ctismer> mattip: With my hack, this part works exactly as before. We don't need any patch tho, we are fine with our `PyMethod_Check` redefinition.

13:15 <charlie_sando> mattip, worked ty!

13:18 <ctismer> mattip: Having PyPy functions pretending to do the same as CPython's, but doing something differently has cost us very much time. Code that used to work suddely stops to work because PyPy has a different understanding of that. It would have been much better for us when PyPy would rename their only 70% COM

13:20 <ctismer> ..compatible functions into something slightly different (like PyPyMethod_Check), because then we would not run into cases which you only can understand if you always work with the compatibility list.

13:42 <mattip> ctismer: the documentation of PyMethod_Check is quite explicit https://docs.python.org/3/c-api/method.html#c.PyMethod_Check

13:42 <mattip> does PyPy not comply with the spec?

13:44 <mattip> I am glad you found a way to get what you wanted

13:44 <mattip> it seems you want a IsBuiltin

13:45 <mattip> or IsOKToCallWithoutTheGil()

13:45 <mattip> although looking at some of the builtins https://docs.python.org/3/library/functions.html

13:46 <ctismer> mattip: The function is completely correct. But PyCFunction is not a method in CPython, but _is_ suddenly in PyPy. Therefore I would either re-build the CPython behavior or rename the function, because it has different scope.

13:46 <ctismer> mattip: Sure this is a weird CPython quirk, agreed.

13:47 <mattip> is it always safe to call a PyCFunction without the GIL?

13:48 <ctismer> mattip: That depends on the function. It is for us, because we know what they are (not) doing.

13:49 <ctismer> mattip: But everything that might call back into the interpreter must hold the GIL.

13:50 <ctismer> mattip: Even a Py_DECREF 🤪

13:53 <mattip> cool. But now I am confused how this relates to PyMethod_Check and PyCFunction

13:53 <mattip> you have a set of functions you know you can treat specially, and you identify them by calling PyMethod_Check ?

13:54 <mattip> which is a generic test for isinstance(types.MethodType)

14:09 <ctismer> mattip: No, vice versa. All the functions where we need to be careful are those which are written in Python. The functions where we have full control are the C functions, which are PyCFunction objects.

14:09 <ctismer> They are the default functions when there is no override. When someone declares a class with an override method, then this comes from Python and needs the GIL. But that are very few, so we can neglect the delay.

14:13 <ctismer> mattip: For instance, `QApplication` has hundreds of methods, one is `notify`. This notify is used heavily in the GUI and written in C. Somemone derives a class from `QApplication` in Python with his own `notify` function. That is written in Python and treated differently.

14:16 <ctismer> mattip: Our problem was only that PyMethod_Check is extended in PyPy to include the cases which are PyCFunction in CPython. That is fixed by now. Of course, a registry would be cleaner, hint taken 😆

14:16 <mattip> ahh, so when calling notify from C, you want to know if it is your notify or one from a user, got it

14:17 <ctismer> mattip: 🖖

14:18 <mattip> could a user override notify via C code with a version that calls back to python, and then it is a PyCFunction, no?

14:19 <mattip> and by chance 98% of users do this in single-threaded code, so it is not critical whether the GIL is held or not

14:20 <mattip> (or at least code where notify and whatever it does is not shared across threads)

14:22 <ctismer> mattip: sure he could. But then he does C code, playing the same game as we do, and he must use the API rules (which are written nowhere, I guess)

14:23 <ctismer> mattip: PySide is written for Python users, oc C programmers who know what they are doing.

16:05 <Hodgestar> ctismer: It sounds like this is a property of the methods one would ideally mark explicitly somehow.

16:09 <ctismer> Hodgestar: Yes. But I'm trying to keep the existing (huge) codebase as it is before doing such rigorous changes, just because PyPy and CPython disagree on something :)

18:18 <LarstiQ> very much feels like a Hyrum's Law situation