<antocuni>
I think that all these bbot2 messages clutter the channels and make it harder to follow dicussions. What about moving them to a dedicated #pypy-bbot channel or so?
bbot2 has quit [Excess Flood]
bbot2 has joined #pypy
<the_rat>
mattip: multiple interpreters in the same process have been a thing in the C-API for a while though. PEP 554 subinterpreters ideally popularizes that. There was quite a bit of effort to move interpreter state away from globals. Of course a lot of C extensions use globals so we are far from universally usable subinterpreters
<mattip>
also modules must use the multiphase initialization API
<mattip>
(so I remembered the problem but in reverse: the problem is in creating static types)
<vstinner>
mattip: and it is a big issue :-/
<vstinner>
mattip: maybe we will consider keep static types in the "main" interpreter, but only use/require heap types in subinterpreters, for backward compatibility
<vstinner>
mattip: this solution is blurry for now :-)
<the_rat>
Yeah, static types are another form of global state
<vstinner>
the root issue for subinterpreters is that putting any kind of lock on PyObject.ob_refcnt quickly becomes a performance bottleneck
<vstinner>
especially for the most commonly used Python objects, like None/True/False, small integer singletons -5..256, empty tuple, etc.
<vstinner>
adding a single if() in Py_INCREF makes CPython ~10% slower <= some people proposed "immortal objects"
<vstinner>
there are other issues, like the magic object.__subclasses__() which lists *all* objects indirectly
<vstinner>
all types, sorry
<the_rat>
There was a suggestion to have refcnts on a per-interpreter table rather than in the object but I don't know how that would look like
<vstinner>
the_rat: Larry Hastings tried many things, and as far as I know, all of his attempts were inefficient
<vstinner>
well, there is no secret: reference couting doesn't scale with the number of threads :-p
<vstinner>
i would prefer to kill refcounting internally, and only use it for the C API
<vstinner>
but well, that's a giant project :)
<vstinner>
*legacy*
idnar has quit []
idnar has joined #pypy
otisolsen70 has joined #pypy
<antocuni>
mattip: build 1015 have ended, so now it's probably a good time to restart the bot
fotis has joined #pypy
fotis has quit [Ping timeout: 244 seconds]
<antocuni>
vstinner: also, to me it's unclear whether heap types are mature enough to be used in the wild. E.g., see my latest email to capi-sig, to which I got no answer
<vstinner>
antocuni: the unclear part is the way to retrieve the module from a heap type, there is an important function which remains private
<vstinner>
antocuni: _PyType_GetModuleByDef()
<vstinner>
antocuni: without _PyType_GetModuleByDef(), it's hard to retrieve the module state in type functions which don't get the "defining type" in an argument
<vstinner>
antocuni: _PyType_GetModuleByDef() is needed if you must not use global variable... like supporting multiphase init API, to support having multiple instances of a module
<antocuni>
vstinner: it's not only that. My question is: how much are heap types actually used in practice? Are we sure that they are ready to be used by default instead of static types? AFAIK the only big project using heap types is pyside, and there are things which are a bit scary
<antocuni>
like bpo 26979 where ctismer basically says "the default tp_dealloc is buggy" but then no definitive conclusion is reached
<vstinner>
antocuni: when i wrote https://vstinner.github.io/isolate-subinterpreters.html : Right now, 43% (89/206) of types are declared as heap types on a total of 206 types. For comparison, in Python 3.8, only 9% (15/172) of types were declared as heap types: 74 types have been converted in the meanwhile.
<vstinner>
antocuni: 89 types are defined as heap types
<vstinner>
antocuni: the problem are types exposed in the C API, like PyLong_Type or PyUnicode_Type
<antocuni>
yes, I know what is the problem
<antocuni>
I'm talking the general robustness of heap types
<antocuni>
like for example, I grepped the top4000 PyPI packages and I couldn't find any heap type which uses a custom tp_dealloc, apart pyside which claims that the default tp_dealloc is buggy
<vstinner>
antocuni: i think Cython still defines static types, no?
<mattip>
how many heap types at all did you find compared to static types
<vstinner>
antocuni: more and more extensions are written with Cython
<antocuni>
yes, so?
<vstinner>
antocuni: so Cython should be updated to produce heap types
<antocuni>
yes
<vstinner>
antocuni: and use the multiphase init API (i don't think that it's already the case)
<antocuni>
and I predict that as soon as people start to use heap types "generally", they will discover lots of bugs and subtle behavior differences
<vstinner>
antocuni: sorry, i didn't see your email about tp_dealloc, it's marked as unread in my inbox, i plan to read it one day :-D
<antocuni>
sure, np :)
<vstinner>
antocuni: CPython has tons of dealloc functions on heap types
<vstinner>
antocuni: not sure what you are talking about
<vstinner>
implementing a type in C is hard. it's easy to miss something
<vstinner>
for example, it's possible to define a heap type without implementation the GC protocol which is wrong
<vstinner>
it's also easy to miss Py_DECREF on the type in the dealloc function
bbot2 has quit [Quit: buildmaster reconfigured: bot disconnecting]
<antocuni>
yes, and that's where my meta-comment generates from. It's likely that since heap types have been used so rarely so far, the more we start to use them the more subtle issues we will find
<vstinner>
"the more we start to use them the more subtle issues we will find" sure
<antocuni>
ok, that's basically all I wanted to say :)
<mattip>
ronan: any thoughts about merging hpy-0.0.3 to py3.7 to get it into the release?
<vstinner>
"[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol"
<antocuni>
thanks for the pointers, I'll try to read them later. I'm especially interested since we are using heap types by default in HPy, and I am worried that people will experience weird bugs because of that
fotis has joined #pypy
<vstinner>
antocuni: this summer, one day, i wanted to start a documentation about all these problems
<vstinner>
antocuni: then i was scared and abandonned the idea
<antocuni>
more docs are always a good idea :)
<vstinner>
i'm still very confused that *all* heap types must define a traverse function
<vstinner>
example:
<vstinner>
{ Py_VISIT(Py_TYPE(self)); return 0; }
<vstinner>
static int lock_traverse(lockobject *self, visitproc visit, void *arg)
<vstinner>
this function fixed a super annoying reference leak in subinterpreters, it took me days to understand it
<Guest27>
otherwise I can just duplicate pypy3.exe as python.exe, but not sure of incidious behavior that can happen in the future, for example a package believing it's standard cpython.
yuiza_ has joined #pypy
fotis has joined #pypy
fotis has quit [Ping timeout: 240 seconds]
<mattip>
Guest27: if you use conda, or bet a nightly, this problem should be solved
<mattip>
we now supply a python.exe, pythonw.exe in the standard zip file