cfbolz changed the topic of #pypy to: #pypy PyPy, the flexible snake | IRC logs: and | insert pithy quote here
lritter has quit [Ping timeout: 252 seconds]
lritter has joined #pypy
stkrdknmibalz has quit [Quit: WeeChat 3.0.1]
lritter has quit [Ping timeout: 252 seconds]
Atque has quit [Quit: ...]
<antocuni> I think that all these bbot2 messages clutter the channels and make it harder to follow dicussions. What about moving them to a dedicated #pypy-bbot channel or so?
bbot2 has quit [Excess Flood]
bbot2 has joined #pypy
<the_rat> mattip: multiple interpreters in the same process have been a thing in the C-API for a while though. PEP 554 subinterpreters ideally popularizes that. There was quite a bit of effort to move interpreter state away from globals. Of course a lot of C extensions use globals so we are far from universally usable subinterpreters
<antocuni> I created a MR for moving bbot2 to another channel:
[Arfrever] has quit [Killed (NickServ (GHOST command used by [Arfreve1]))]
[Arfrever] has joined #pypy
<mattip> antocuni: sounds good
<antocuni> mattip: I think we need to restart the builbot master and/or to reload the config in order to actually move the bot?
<mattip> right, I am on it
<antocuni> thanks
<mattip> there is one build running, I will restart when it finishes
<antocuni> 👍
<mattip> the_rat: in addition to the global state problem, I seem to recall a problem with creating non-static types
<mattip> but maybe that got cleaned up?
<antocuni> mattip: what was the problem with heap types?
<mattip> says that all the CPython static types must be converted to heap types for subinterpreters
<mattip> also modules must use the multiphase initialization API
<mattip> (so I remembered the problem but in reverse: the problem is in creating static types)
<vstinner> mattip: and it is a big issue :-/
<vstinner> mattip: maybe we will consider keep static types in the "main" interpreter, but only use/require heap types in subinterpreters, for backward compatibility
<vstinner> mattip: this solution is blurry for now :-)
<the_rat> Yeah, static types are another form of global state
<vstinner> the root issue for subinterpreters is that putting any kind of lock on PyObject.ob_refcnt quickly becomes a performance bottleneck
<vstinner> especially for the most commonly used Python objects, like None/True/False, small integer singletons -5..256, empty tuple, etc.
<vstinner> adding a single if() in Py_INCREF makes CPython ~10% slower <= some people proposed "immortal objects"
<vstinner> there are other issues, like the magic object.__subclasses__() which lists *all* objects indirectly
<vstinner> all types, sorry
<the_rat> There was a suggestion to have refcnts on a per-interpreter table rather than in the object but I don't know how that would look like
<vstinner> the_rat: Larry Hastings tried many things, and as far as I know, all of his attempts were inefficient
<vstinner> well, there is no secret: reference couting doesn't scale with the number of threads :-p
<vstinner> i would prefer to kill refcounting internally, and only use it for the C API
<vstinner> but well, that's a giant project :)
<vstinner> *legacy*
idnar has quit []
idnar has joined #pypy
otisolsen70 has joined #pypy
<antocuni> mattip: build 1015 have ended, so now it's probably a good time to restart the bot
fotis has joined #pypy
fotis has quit [Ping timeout: 244 seconds]
<antocuni> vstinner: also, to me it's unclear whether heap types are mature enough to be used in the wild. E.g., see my latest email to capi-sig, to which I got no answer
<vstinner> antocuni: the unclear part is the way to retrieve the module from a heap type, there is an important function which remains private
<vstinner> antocuni: _PyType_GetModuleByDef()
<vstinner> antocuni: without _PyType_GetModuleByDef(), it's hard to retrieve the module state in type functions which don't get the "defining type" in an argument
<vstinner> antocuni: _PyType_GetModuleByDef() is needed if you must not use global variable... like supporting multiphase init API, to support having multiple instances of a module
<antocuni> vstinner: it's not only that. My question is: how much are heap types actually used in practice? Are we sure that they are ready to be used by default instead of static types? AFAIK the only big project using heap types is pyside, and there are things which are a bit scary
<antocuni> like bpo 26979 where ctismer basically says "the default tp_dealloc is buggy" but then no definitive conclusion is reached
<vstinner> antocuni: when i wrote : Right now, 43% (89/206) of types are declared as heap types on a total of 206 types. For comparison, in Python 3.8, only 9% (15/172) of types were declared as heap types: 74 types have been converted in the meanwhile.
<vstinner> antocuni: 89 types are defined as heap types
<vstinner> antocuni: the problem are types exposed in the C API, like PyLong_Type or PyUnicode_Type
<antocuni> yes, I know what is the problem
<antocuni> I'm talking the general robustness of heap types
<antocuni> like for example, I grepped the top4000 PyPI packages and I couldn't find any heap type which uses a custom tp_dealloc, apart pyside which claims that the default tp_dealloc is buggy
<vstinner> antocuni: i think Cython still defines static types, no?
<mattip> how many heap types at all did you find compared to static types
<vstinner> antocuni: more and more extensions are written with Cython
<antocuni> yes, so?
<vstinner> antocuni: so Cython should be updated to produce heap types
<antocuni> yes
<vstinner> antocuni: and use the multiphase init API (i don't think that it's already the case)
<antocuni> and I predict that as soon as people start to use heap types "generally", they will discover lots of bugs and subtle behavior differences
<vstinner> antocuni: sorry, i didn't see your email about tp_dealloc, it's marked as unread in my inbox, i plan to read it one day :-D
<antocuni> sure, np :)
<vstinner> antocuni: CPython has tons of dealloc functions on heap types
<vstinner> antocuni: not sure what you are talking about
<vstinner> implementing a type in C is hard. it's easy to miss something
<vstinner> for example, it's possible to define a heap type without implementation the GC protocol which is wrong
<vstinner> it's also easy to miss Py_DECREF on the type in the dealloc function
bbot2 has quit [Quit: buildmaster reconfigured: bot disconnecting]
<antocuni> well, the TL;DR version of my mail is: what is the official answer to this comment?
<antocuni> is the default tp_dealloc correct or is it wrong?
<mattip> could someone chage the topic for the channel to mention "buildbot is on #pypy-bbot"
<Dejan> I do not mind bbot being here!
<antocuni> I think cfbolz has the power to do it ^^^
<antocuni> Dejan: too late :)
* antocuni tries to start a build to check if it works
<antocuni> cool, it seems to work
<vstinner> antocuni: i didn't know it seems to contain multiple sub-issues
<antocuni> yes, and that's where my meta-comment generates from. It's likely that since heap types have been used so rarely so far, the more we start to use them the more subtle issues we will find
<vstinner> "the more we start to use them the more subtle issues we will find" sure
<antocuni> ok, that's basically all I wanted to say :)
<mattip> ronan: any thoughts about merging hpy-0.0.3 to py3.7 to get it into the release?
<vstinner> antocuni: was a super complex bug
<vstinner> antocuni: mmmh, maybe i was thinking at
<vstinner> "[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol"
<antocuni> thanks for the pointers, I'll try to read them later. I'm especially interested since we are using heap types by default in HPy, and I am worried that people will experience weird bugs because of that
fotis has joined #pypy
<vstinner> antocuni: this summer, one day, i wanted to start a documentation about all these problems
<vstinner> antocuni: then i was scared and abandonned the idea
<antocuni> more docs are always a good idea :)
<vstinner> i'm still very confused that *all* heap types must define a traverse function
<vstinner> example:
<vstinner> { Py_VISIT(Py_TYPE(self)); return 0; }
<vstinner> static int lock_traverse(lockobject *self, visitproc visit, void *arg)
<vstinner> this function fixed a super annoying reference leak in subinterpreters, it took me days to understand it
<vstinner> and i created to explain that *all* heap types have the bug, not only _thread.Lock
<vstinner> antocuni: in terms of documentation, the PEP 630 is good
<antocuni> it also means that all heap types should have Py_TPFLAGS_HAVE_GC, but I'm not sure it's the default
<vstinner> not "should" but "must"
<vstinner> IMO it should be an error, to create a heap type without Py_TPFLAGS_HAVE_GC or with no traverse function
<antocuni> yes
<vstinner> raise an exception
<antocuni> this is the kind of subtle issues I'm talking about
<vstinner> but it may break tons of C extensions, i didn't check
<vstinner> antocuni: hey, it's funnier if it's not documented and there is no warning!
<vstinner> antocuni: more seriously, nobody proposed to write such doc yet
<vstinner> antocuni: the requiring for GC/traverse was only only fixed very recently (in 3.10 which is not released yet, RC1 was released)
krono has quit []
krono has joined #pypy
fotis has quit [Ping timeout: 256 seconds]
fotis has joined #pypy
<ronan> mattip: it's not ready quite yet but I'll try to finish it ASAP
<mattip> ok thanks. fwiw it translates
fotis has quit [Ping timeout: 252 seconds]
fotis has joined #pypy
fotis has quit [Ping timeout: 244 seconds]
ronan has quit [Ping timeout: 240 seconds]
ronan has joined #pypy
Guest27 has joined #pypy
<Guest27> to reduce the difference of behavior between python.exe and pypy3.exe from python-vscode, may I suggest you "upvote" here ?
<Guest27> otherwise I can just duplicate pypy3.exe as python.exe, but not sure of incidious behavior that can happen in the future, for example a package believing it's standard cpython.
yuiza_ has joined #pypy
fotis has joined #pypy
fotis has quit [Ping timeout: 240 seconds]
<mattip> Guest27: if you use conda, or bet a nightly, this problem should be solved
<mattip> we now supply a python.exe, pythonw.exe in the standard zip file
<Guest27> I don't use conda, I use the standard "pip" package manager
<Guest27> ah, ok . that's great
<Guest27> downloaded 1 hour ago, but didn't look. so that's solves the problem of decision for me. thanks
stkrdknmibalz has joined #pypy
jacob22 has quit [Ping timeout: 256 seconds]
yuiza has quit [Quit: Leaving.]
yuiza_ has quit []
yuiza_ has joined #pypy
Guest27 has quit [Quit: Connection closed]
otisolsen70 has quit [Quit: Leaving]
jacob22 has joined #pypy
yuiza_ has quit [Quit: Connection closed for inactivity]
lritter has joined #pypy
fotis has joined #pypy
fotis has quit [Ping timeout: 252 seconds]