Techcable_ has quit [Quit: ZNC - https://znc.in]
Techcable has joined #hpy
mattip has quit [*.net *.split]
_0az3 has quit [*.net *.split]
pmp-p has quit [*.net *.split]
_0az3 has joined #hpy
mattip has joined #hpy
pmp-p has joined #hpy
<fijal> vstinner: can we have a PEP that says "CPython should export less than 1024 symbols"? ;-)
<vstinner> fijal: we are at 1080 :-( i need to remove a few more
<vstinner> fijal: at the same time, other people love to add new functions
<vstinner> fijal: i plan to add tons of functions to abstract access to structures
<fijal> yeah I know
<fijal> I really appreciate your effort even if I call you Don Quixote every now and again ;-)
<vstinner> fijal: i would like to make PyThreadState opaque. it has 38 members. well, some members are really for internal purpose
<vstinner> but members which are useful in 3rd party code, they may need 2 functions per member, getter and setter
<fijal> how is hpy doing? I have been out of the loop for a while
<vstinner> fijal: about the PEP idea: i proposed 4 or 5 PEPs, most of them as draft. i got no feedback or bad feedback
<vstinner> fijal: most people are strong supporters of "don't touch anything"
<vstinner> fijal: or "move, there is no problem, look somewhere else"
<vstinner> i started from Hodgestar's HPy Manifesto and wrote a PEP saying that "hey, the C API has flaws, and there are reasons why it's hard to fix them"
<vstinner> but the feedback that i got is "what is the point of this PEP"
<vstinner> and honestly, i'm not sure myself if the PEP has any purpose :-D
<fijal> so CPython devs still have no idea why having multiple python implementations is a good idea?
<fijal> are they also convinced fast python is a non-goal?
<vstinner> fijal: are you surprised?
<vstinner> fijal: "Python performance" was Python motto for a long time :-)
<fijal> I'm just pondering
<vstinner> fijal: "suddently", Instagram came with 2 new Python implementations, and now Microsoft is investing tons of money on it
<fijal> it seems to me, like mitsuhiko pointed out, that python developers and users are not living in the same part of the planet
<vstinner> it's just that Oracle is "investing" on HPy, to make GraalPython successful
<fijal> we need some buy-in, I guess
<vstinner> fijal: i heard that feeling about CPython core devs :-)
<fijal> well, that would be yet another example
<fijal> where everyone is talking about python performance, but core devs are like "pff no, why would you want that"
<vstinner> fijal: people saying "backward compatibilty matters" are right, people saying "python performance matters" are right. but both goals are somehow exclusive
<fijal> "backwards compatibility"
<fijal> well, same people change things all the time, so I'm not too sold
<fijal> like "python 3" *cough*
<vstinner> many parts of Python are inefficient "by design"
<vstinner> fijal: because of python 3, people are even more afraid of an hypothetical "python 4" which would split the community again
<fijal> I mean specifically being openly hostile to the concept of non-cpython implementations
<vstinner> that's why we cannot break the whole C API in a flag day
<fijal> sure, but I don't think anyone is asking about that?
<fijal> "stop adding new features" is not breaking it
<fijal> or "let's think about refcounting a bit harder" etc.
<vstinner> fijal: all problems we are talking about are hard problem, it costs a lot of working hours and a lot of money to fix them
<fijal> yeah of course
<vstinner> fijal: nobody wants to work on that, and prefer to focus on low hanging fruits
<fijal> but it's even harder if someone is actively preventing you from doing work
<fijal> which is my experience with cpython devs - they really don't want to make it easier for other implementations
<fijal> because it restricts what they can do
<vstinner> fijal: and you're right
<vstinner> and they are right :)
<fijal> yeah I understand the motivation, sure
<vstinner> i'm trying to dig a little bridge between the two worlds
<vstinner> oops, build
<vstinner> not dig :-D
<vstinner> fijal: it seems like the shift must come from CPython devs
<fijal> yeah
<vstinner> fijal: requests for supporting other Python implementations done by people outside CPython was not taken in account seriously, you're right
<fijal> the shift might be "eh, those guys know something about performance, maybe we should listen"
<vstinner> fijal: my strategy is to explain the problem this way: CPython has flags, we must fix CPython
<fijal> flags?
<vstinner> fijal: to fix CPython, we must hide implementation details
<vstinner> fijal: sorry, flaws*
<fijal> it's a hard story to sell, usually
<vstinner> fijal: fix things like exposing ref counting as part of the C API (borrowed ref, steal references), API which prevent to move objects in memory, etc.
<vstinner> fijal: things are changing
<fijal> I mean, I would love a python language committee that would really put brakes on adding new features
<fijal> I'm pretty sure this is a very unwelcomed idea
<vstinner> fijal: "put brakes on adding new features" as discussed a few years ago, the idea was quickly rejected
<vstinner> this is not going to happen
<vstinner> you need to find other ways to make the life simpler for other Python implementations
<fijal> I presume this is then a no-go
<fijal> "python refuses to grow up" kinda vibe
<vstinner> so far, what i did is to identify bad patterns in the C API, and first attempt to avoid them in C API additions
<vstinner> https://devguide.python.org/c-api/ now requires to use strong references, for example
<vstinner> fijal: thinking about a sudden major shift in Python is worthless. i prefer to focus on what worked previously: incremental changes
<fijal> "worked"
<vstinner> fijal: for example, fix 1 or 2 issues in the C API is fine. removing the C API is not :)
<fijal> sure, but "10 years into the effort, we slowed down the addition of new C APIs a bit" is not progress at all, in my eyes
<vstinner> my worry is that i still see a huge gap between the current C API and HPy
<vstinner> i'm trying to bend the C API towards HPy
<vstinner> I added Py_NewRef() and Py_Is() for example
<vstinner> "incremental changes" :-)
<vstinner> my plan is to make the transition to HPy or a HPy-like API smoother
<vstinner> otherwise, HPy will be abandonned at all previous attempt to "fix" or "enhance" Python
<vstinner> Python has a very big cemetery of abandonned projects :-(
<vstinner> Pyjion, Pyston, Unladen Swallow, etc. ... Pyston is back to life :-)
<vstinner> fijal: CPython has weird constraints like continuing to support abandonned (unmaintained) C extensions as long as possible
<vstinner> fijal: also, it's possible to require a few minor changes once a year (at each Python release), it's not possible to require to require big C extensions
<vstinner> otherwise, i'm asked to revert my changes :-p
<vstinner> so far, i made steady progress. i fixed many things in the C API. i also prepared the API for incompatible changes. for example, in Python 3.11, it's now way simpler to make PyObject opaque. even if today it is not possible
<Hodgestar> Wow. Much text. :)
* Hodgestar gets coffee and reads.
<vstinner> Hodgestar: hey :)
<Hodgestar> A quick reminder that at the language summit, everyone was actually pretty supportive of HPy.
<Hodgestar> So it's not all doom and gloom.
<Hodgestar> From an HPy perspective, the C API doesn't need to be bent towards HPy.
<Hodgestar> But it is still very useful to HPy for many people (and especially people working on important relevant projects like CPython and Cython) to understand why HPy is desirable.
<Hodgestar> The C API is a hinderance to CPython's own evolution.
<Hodgestar> And bending the C API towards HPy would allow CPython itself to become better.
<Hodgestar> I am certainly sympathetic to the negative view of "getting the C API opaque enough by evolutionary changes would take longer than our lifetimes".
<Hodgestar> For me the point of a PEP is to get more community agreement that HPy is a part of the way forward so that once HPy is more mature, we have already started the journey of getting it adopted.
<mattip> "community agreement" may be too strong, I would settle for "community awareness".
<mattip> I think once we can get HPy into cython/pybind11, and help core libraries like NumPy mostly use it, it will become the normative way to do things
<mattip> the big blocker right now, from my perspective, is using heap-allocated types with meta classes and finalizers
<mattip> it would be nice if that would be part of CPython 3.11
<vstinner> "The C API is a hinderance to CPython's own evolution" yes :)
<vstinner> mattip: finalizers are supported with heap types, you can implement tp_dealloc no?
<vstinner> mattip: are you talking about issues with subclasses?
<vstinner> mattip: is the problem the usage of finalizers *and* meta classes?
<mattip> dunno. Whatever that long-open issue was about and antocuni asked about on the c-api list
<vstinner> mattip: alright, thanks
<mattip> types from spec, not necessarily heap types
<mattip> well both maybe, I dunno. The whole thing is confusing and worrisome
<mattip> this is a blocker to use the limited API
<mattip> FWIW, this is an example of the cost of maintenance vs. cost of new,
<mattip> it is much easier to add new functions even if they are wrappers
<mattip> than to fix/maintain older APIs
<vstinner> i'm writing an article about adding abstractions to the PyObject structure. is it correct to say that HPy allows to allocate an object on stack and copy it the heap memory when it's added to a container: on HPy_Dup()?
<vstinner> i'm trying to explain that the C API prevents to allocate temporarily objects on the stack
<mattip> it would also be nice to have a fix for https://bugs.python.org/issue15870 by 3.11
<vstinner> mattip: someone has to write a PR, someone else has to review and merge it ;-) i'm not available for that
<mattip> :(
<mattip> there is no consensus on an accepted API, so it is too early to submit a PR
<mattip> For the first issue, it is not clear what is supposed to happen
<mattip> for the second, no core dev has supported any of the options in the comment
<phlebas> vstinner: Just to throw in my 2c - I think (also on the quick python-dev discussions around this topic) that people are always throwing too many things together. "but there are gimp plugins that are not on PyPI". "There are unmaintained projects" etc
<phlebas> so I think as long as you cannot delineate the goals more clearly, people come from all sorts of directions with arguments agains
<phlebas> and you're changing things that may not have to change right now means those unmaintained extensions and the puported "long tail" of extensions not on PyPI is left behind. so I feel that pushing Cython and HPy hard for new developments and active projects is the better option. the active projects will benefit from an abstraction level that lets them run efficiently on alternate impls.
<phlebas> and old projects, well, they either do or don't continue to run on cpython
<fijal> it's not like old projects work with new pythons either
<phlebas> fijal: true. but i understand that people see a difference between "got broken to get this new feature in" and "got broken for the sake of making the C API leaner"
<fijal> I don't, I think
<phlebas> well, if your extension got broken that you use but don't really enjoy working on, and now you need to fix it, you ask yourself: why was it broken? if the answer is "for this cool new CPython feature, go fix your extension and enjoy" - that's a good answer to me. but the answer "because we're changing the C API. you know. for the FUTURE!" - a frustrating answer
<phlebas> from that POV, I'd personally prefer if someone wants to fix things, break everything bad once, so I need to migrate once. but the Py3 experience seems to have put core off
<phlebas> hence my leaning towards HPy and Cython. It's work to migrate there, but the majority is work once, and the potential stability much better, since now HPy and Cython will (to some degree) insulate me from CPython API changes
<vstinner> phlebas: i agree that C extensions should use Cython, HPy or any other "abstraction". Cython is recommended by the Python C API
<phlebas> yeah, and fangerer is making good progress with the HPy backend for Cython, so that would help alt impls also
<vstinner> mattip, Hodgestar : i wrote an article explaining why I want to make the PyObject structure opaque: https://vstinner.github.io/c-api-abstract-pyobject.html
<vstinner> in short, it prevents a wide range of optimizations
<vstinner> making PyObject opaque alone doesn't suddenly unlock these optimizations, but it moves CPython closer to that ;-)