cfbolz changed the topic of #pypy to: #pypy PyPy, the flexible snake | IRC logs: and | hacking on TLS is fun, way more fun than arguing over petty shit, turns out
danchr_ has quit [*.net *.split]
indyZ has quit [*.net *.split]
danchr_ has joined #pypy
indyZ has joined #pypy
Atque has quit [Remote host closed the connection]
<fijal> cfbolz: largely just tests being sloppy so far w.r.t. c99
<fijal> I got into a mess of installing libgc though, why do linkers have to be so complex....
Guest96 has joined #pypy
xcm_ has quit [Remote host closed the connection]
xcm_ has joined #pypy
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
otisolsen70 has joined #pypy
alice has quit [Remote host closed the connection]
jstoker has quit [Remote host closed the connection]
alice has joined #pypy
jstoker has joined #pypy
Guest96 has joined #pypy
<fijal> mattip: do we need native homebrew in an unusual location? this will be quite a nightmare to deal with
<mattip> dunno, I just followed the guides I found
<fijal> ok
<fijal> can I have ack-grep too?
<mattip> brew install ack-grep -> error
<mattip> brew install ack succeeded
<mattip> is that good enough?
<fijal> yes, thank you
<fijal> the amount of shit I need to fight with to get the development environment going is annoying :/
marvin_ has quit [Remote host closed the connection]
marvin has joined #pypy
<fijal> mattip: ok, so the current problem is pkg-config bdw-gc gives us garbage
<fijal> and I would imagine this would be a problem for everything, since the x86_64 is the default path, can we somehow swap them?
<mattip> can't you change the PATH to find /opt/homebrew/bin/pkg-config if arch == 'arm64'
<fijal> that's almost certainly a problem
<fijal> it seems to me like having by default the non-native architecture would create issues, always
<fijal> seems like I'm adding hacks for this specific layout, that's probably a bit bad
<fijal> what do we need x86 homebrew for?
<fijal> If you’ve been using Homebrew under emulation, you now have two side-by-side installs, so from this point on it’s a matter of working tool by tool, installing an ARM64 versions then removing the x86-64 version
<fijal> so yes, they suggest removing the x86_64 one once we have one for arm64
<fijal> mattip: have you found out why I can't use homebrew myself?
<mattip> "groups matti" and "groups fijal" are equivalent, so no idea
<mattip> what happens when you try
<fijal> ah ok, so I need sudo
<fijal> but I don't have a password
<fijal> (I never did)
<mattip> hmm, I wonder how I am different
<fijal> you don't have to use sudo?
<mattip> nope
<fijal> ok, well, the entire /usr/local belongs to matti
<LarstiQ> has the group been added after fijal logged in? (i.e. relogin or start a new shell)
<fijal> matti/admin
<LarstiQ> ah, or that :)
<mattip> ok, so make it admin/admin
<mattip> feel free to wipe /usr/local and start over
<fijal> ok
<fijal> well, it still belongs to matti
<fijal> so I can't :)
<fijal> (I think I'm ok uninstalling packages one by one as they say)
<Guest96> you can just nuke brew and do the arm install
<Guest96> although I have both x86 and arm brew side by side for $reasons, which works currently, but will go wrong
<mattip> it must be some difference with permissions. "chown admin.admin" fails of course since admin is a group
<Guest96> I don't know of this helps, but here /usr/local belongs to 'system' with rw, and 'wheel' has r and 'everyone' has r, but the brew dirs in /usr/local belong to my actual user
<Guest96> of / if /s
<Guest96> I know very little about macOS permissions apart from "try not to change the defaults" and "be cautious with sudo"
<fijal> mattip: maybe make it fijal:admin then?
<mattip> ok, I guess we can change it back as needed
<mattip> jsut one more check: I did "chmod -R g+w", did that change anything?
<fijal> nope
<fijal> ls -lh /usr/local/var/homebrew/locks
<mattip> one more time (this time I did chmod with sudo)
<fijal> better
<fijal> mattip: almost?
<fijal> I think I lost access to Cellar
<fijal> thans for helping with this
<mattip> ok, that was weird. Somehow everyone lost "r" permissions. Try now?
<fijal> much better, uninstalled the gc
<mattip> so if you want to, go ahead and figure out how to remove x86_64 brew and put arm64 brew on top
<fijal> ok, now same with /opt/homebrew I think
<mattip> if I need x86_64 brew, I will re-install it to /opt/ something
<fijal> I think I'll try to do what they say, remove one by one
<mattip> chmod on /opt/homebrew is done
<mattip> can you share the link?
<mattip> to "they say"
<mattip> ahh, I missed that "it’s a matter of working tool by tool, installing an ARM64 versions then removing the x86-64 version"
<mattip> bdw-gc is already in /opt/homebrew, so no need to reinstall
<fijal> I wonder why pkg-config does not pick it up
<fijal> mattip: ok, we probably want the arm stuff in /usr/local I would imagine indeed
<mattip> dunno. We do want other people to be able to replicate this, so we should be "standard", but I have no idea what that means for M1
<mattip> maybe worth asking someone somewhere
<fijal> I don't know, but I would imagine it would be standard to not have x86_64 binaries very soon
<fijal> can we just do that so I can progress?
<mattip> sure, go ahead
<mattip> do I need to wipe something clean or can you do it?
<fijal> I don't know
<fijal> should I just nuke /usr/local and try again?
<fijal> hm, wait
<fijal> ok, maybe it's just a matter of shifting PATH
<fijal> ok, test passed! yay!
<fijal> mattip: thank you!
<mattip> cool, good luck
<fijal> test/
<fijal> is a whack test to explode
<Guest96> afaict the situation on m1 doesn't have 'standard' yet, I switched about 4 weeks
<Guest96> ago
<Guest96> I think in lots of projects the mac maintainers have been waiting until this m1 Pro was available
<mattip> Guest96: switched? from what to what?
<Guest96> intel mac -> arm
Guest96 is now known as andythenorth
<andythenorth> if the binary is an actual double-click mac app bundle then it's likely to be a Universal build (this is what OpenTTD did)
<andythenorth> I think we're all hoping the brew ports tree packages just switch to arm builds asap
<mattip> it seems at least scientific python packagers are hesitant to provide universal2 builds
<mattip> there are some problems with "sizeof(long long)" and some other strange corner cases
<andythenorth> super :)
<mattip> maybe like fijal says, x86_64 will slowly just go away
<andythenorth> that's my expectation
<andythenorth> there's precedent with the end of 32 bit support and the prior PowerPC -> intel switch
<andythenorth> most stuff gets done in about a year, then there are always a few critical things everyone uses it that take longer, but get done eventually
<andythenorth> my English today is terrible sorry, I am a native English speaker, just have no brain without cofee
<cfbolz> andythenorth: don't worry, we all aren't
<andythenorth> at least endian-ness no longer seems to be the issue it was about 20 years ago :P
<fijal> pffff
<fijal> ok I found the first problem
<fijal> ctypes on python does not really use variadic arguments correctly?
<mattip> this is a libffi issue right? I wonder if I should update to macos12 on the machine?
<fijal> cpython ships it's own libffi right?
<fijal> I think I'll skip the test for now
<fijal> maybe add a hack for open at some stage, seems like an interesting function
<fijal> pom pom pom
<fijal> mattip: is there a way to update libffi somehow?
<fijal> I wonder what is cpython using
<fijal> ok.... so libffi on python3 works, the one on python2 does not
* fijal double checks
<fijal> indeed
<fijal> mattip: ideas?
<fijal> skipping all the llinterp tests that do "open" seems a bit harsh
<fijal> maybe there is a better strategy than calling open via ll2ctypes, but ll2ctypes is a major mess
<cfbolz> fijal: ll2cffi!
<fijal> seems a bit excessive for this branch.... we thought about it!
<mattip> where does cpython3 get their arm64 libffi from? Do they build from source?
<mattip> we could rebuild a python2 with that libffi
<mattip> or use pypy2.7, it should already be updated
<mattip> fwiw, the windows buildbot already uses pypy2.7 not cpython2.7
<mattip> ahh, no, wait, you need a macOS-arm64 python2.7, so pypy wouldn't work yet
<cfbolz> mattip: seems it really needs to be a mac-specific libffi
<cfbolz> they have a different calling convention for variadic functions
<mattip> I am pretty sure the libffi people fixed, that with the latest release, the problem is no-one is building cpython2.7
<cfbolz> yes, seems very plausible
<fijal> mattip: ok, I think if we can get a custom built libffi for our cpython, that would be good
<mattip> there is a brew recipe for libffi, is that good enough?
<fijal> I doubt python will use that?
<fijal> it's already there for example
<mattip> ahh, so you need to compile from source
<mattip> there is a recipe in the buildbot Dockerfile, it is one line (once you download the sources)
<mattip> ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib" && make && make altinstall
<mattip> that is what is used on the aarch64, x86_64, and i686 buildbots
greedom has joined #pypy
<fijal> for python?
<fijal> I guess I need to use /opt/homebrew, but otherwise should work
<fijal> mattip: is everything needed installed there or would I get python with half the stuff?
<fijal> how does one pull python2.7 from somewhere?
otisolsen70 has quit [Quit: Leaving]
<fijal> mattip: ok, so a serious complication - libffi shipped with python is too old
<fijal> we probably should replace that libffi with something newer
<fijal> but that's like hacking cpython source code (maybe fine?)
<fijal> I don't think making it use system libffi would help a whole lot
<mattip> the buildbots get python from
<mattip> they pre-install a bunch of things, AFAICT they are all needed for PyPy too
<mattip> including an up-to-date libffi
<mattip> ahh, wait, python ships a libffi for 2.7?
<mattip> yeah, maybe this is needed
<mattip> nope, that is all about windows
<fijal> I'm pretty sure we need to pull libffi from somewhere else (maybe just copy-paste it here)
<fijal> yeah it's a mess
<fijal> meh
<mattip> I don't understand how cpython3.9 works, I don't see any changes in years in Modules/_ctypes/libffi_osx that would have helped macOS-arm64
<mattip> "use system libffi instead of vendored one on Mac OS 10.15 and up, use correct ABI for variadic functions on arm64, check for libs in the dyld shared cache"
<ctismer> mattip: might this be related to my problems on macOS?
<mattip> are you using M1 Apple silicon arm64?
<ctismer> no
<mattip> so no :)
<ctismer> ah :)
<mattip> well, with that who knows :shrug:
<mattip> ctismer: btw, I started addng a test for PyCFunction_New, but got stuck
<ctismer> mattip: Anyway, after very long hacking, I again reached a point where I don't understand the world. I set an attribute on an instance, I can check it with `PyObject_HasAttr`, but it is not found in `obj->ob_dict. With CPython, it is.
<mattip> what interface do you use to set the attribute?
<ctismer> mattip: I will now really send you a set of PyPy and Python wheels, to have a look. 🤪
<mattip> cool
<ctismer> mattip: one set...
<ctismer> sec
<ctismer> mattip: The setattr is called, but nothing lands in the dict
<ctismer> mattip: This turned out to be the problem: I even can check with `PyObject_HasAttr`, but the according dict is still not touched.
* ctismer is at least _much_ more eloquent in `lldb` now :)
<fijal> mattip: ok, let me know if you find out how to do it on python 2.7, I'm making some food
<mattip> so the obj->ob_dict might be reflected only once when the pyobj is set up, and not updated for the added w_obj.w_dict field?
<mattip> shouldn't all calls should go through the c-api and not directly access obj->ob_dict?
<ctismer> mattip: can be, yes. But where does it get the `PyObject_HasAttr` result if no object is set. Is that a cache effect?
<mattip> there are two objects: the c-level pyobj and the rpython w_obj
<ctismer> !
<mattip> dunno what guarentees we give about synchronizing their state
<mattip> for instance, PyListObject is totally opaque, you cannot get anthing interestng out of the pyobj
<ctismer> mattip: we are fiddling with ob_dict quite often. I thought we are safe when we are not modifying types?
<mattip> if you expect the changes in ob_dict to be synced to the w_obj, I think you will be dissapointed
<ctismer> mattip: yes, that's ok (list object). But SbkObject is our own stuff, that is not clear
* ctismer building wheels, anyway
<mattip> once the w_obj is created from the pyobj, or the pyobj is created from the w_obj, the w_obj dict and the obj->tp_dict go their separate ways
<mattip> so if you manipulate the obj->tp_dict directly from C, you will only see those changes when directly using the obj_tp_dict
<mattip> all the C-API calls and python code will use w_obj.__dict__
<ctismer> mattip: aha! (Grübel...) What would be the right way to do what we do...
* ctismer should have asked this 3 weeks earlier 🙃
vstinner has joined #pypy
<vstinner> hello. i wrote
<vstinner> PEP 674 -- Disallow using macros as l-value
<LarstiQ> ctismer: how often do you do that? Can you guarantee all the changes are done before the other object is created?
<ctismer> vstinner: that is great!
<vstinner> hello. i wrote PEP 674 -- Disallow using macros as l-value which says that it will help PyPy cpyext to avoid converting PyPy objects to CPython objects in the long term:
<vstinner> do you think that this statement is correct?
<vstinner> for example, "Py_SIZE(obj) = new_size" must now be replaced with "Py_SET_SIZE(obj, new_size)"
<vstinner> my assumption is that supporting "Py_SIZE(obj) = new_size" requires to expose the PyVarObject structure and convert a PyPy object to a CPython object
<vstinner> whereas it might be possible to implement Py_SET_SIZE(obj, new_size)" on the PyPy object directly
<vstinner> maybe Py_SET_SIZE() is a bad example
<ctismer> LarstiQ: I don't know exactly. Parts of the code reach back to 2009, I wrote maybe 10 percent or less...
<mattip> AFAICT in order to call Py_SET_SIZE in any form (macro, function) a PyObject* must exist. So the conversion has to happen no matter how you frame it
<ctismer> LarstiQ: So to understand this fully: Direct ob_dict access is problematic, also with my self-created objects?
<vstinner> mattip: ah? there is no way to change a list size in PyPy for example?
<ctismer> mattip: to make sure: I am talking of `obj->ob_dict``, not `type->tp_dict`
<ctismer> mattip: to make sure: I am talking of `obj->ob_dict`, not `type->tp_dict`
<mattip> vstinner: as far as I can tell we don't support Py_SET_SIZE at all
<ctismer> LarstiQ: So, we have `->ob_dict` 16 times in the code, in 4 files. Oe is a conerator.
<mattip> at least it is not anywhere in our codebase
<mattip> ctismer: we do not have any ob_dict in our codebase either, so I am not sure what you are setting :)
<mattip> maybe my grep foo is broken today
<vstinner> mattip: so do you think that the section is wrong and should be removed?
<vstinner> "This PEP alone is not enough to get rid of the CPython objects in the PyPy cpyext module, but it is a step towards this long term goal. PyPy already supports HPy which is a better solution in the long term." :-p
<vstinner> this sentence is based on the previous section saying that the PEP should help migration C extensions to HPy
<ctismer> mattip: we create the objects this way, for instance
<ctismer> mattip: this ob_dict is _why_ a problem? This is pure C code.
<cfbolz> ctismer: two questions: is this dict *only* for c code?
<cfbolz> or should python see these attributes?
<cfbolz> if you rename the attribute to not ob_dict, you would need suitable __getattribute__ implementations, right?
<ctismer> cfbolz: this was not clear to me, but right, that's a special name. Have to do a good analysis. But great that I understand now !
<cfbolz> ctismer: I think what is happening is that you manage to store the attribute into the rpython dict, which is why it's seen by hasattr, but not synced to ob_dict
<ctismer> cfbolz: so if I want to control these attributes via dict, I should either rename them (private only), or I need to use `obj.__dict__` ?
<fijal> hm
<fijal> I wonder if I'm getting stuck on unrelated failures
<ctismer> cfbolz: that explains a lot 😀
<cfbolz> ctismer: yes, either you rename the C field and implement a getattribute etc
<ctismer> cfbolz: mattip: LarstiQ: You made my day (actually 3 weeks)
<cfbolz> ctismer: I wonder though:
<cfbolz> maybe it is enough to read ob_dict some other way
<cfbolz> there is probably some function that reads .__dict__, no?
<cfbolz> PyObject_GenericGetDict?
<cfbolz> so it might be worth a try to replace ->ob_dict with a call to that
Julian has joined #pypy
Julian has quit [Ping timeout: 252 seconds]
Julian has joined #pypy
<ctismer> cfbolz: yes, of course. I will try. Great, that's progress, already.
<ctismer> cfbolz: BTW. `ob_dict` is not found in the PyPy documentation (keyword search)
<cfbolz> ctismer: yes, we don't really have a lot of good info about cpyext
Julian has quit [Ping timeout: 252 seconds]
<ctismer> cfbolz: this is not complaining, I just made sure that I don't forget something knowable
<cfbolz> ctismer: I think you are in unexplored land here ;-)
Julian has joined #pypy
* ctismer thinks there is no better way to learn about PyPy than to port a huge library 😅
<vstinner> mattip: i wrote to remove the PyPy section. thanks for your feedback ;-)
<ctismer> cfbolz: we can augment the docs by using my PYSIDE-535 comments...
<cfbolz> ctismer: yes, we should
<ctismer> cfbolz: so if I use `PyObject_GenericGetDict` to touch the `ob_dict`, then my `PyDict_SetItem` will land where they should? Probably nothing that is done frequently
<mattip> ctismer: why not use the PyObject_SetAttr* c-api functions?
vstinner has left #pypy [#pypy]
<ctismer> mattip: not sure why they did that. Would the PyObject_GenericGetDict not work? It has the advantage of no semantic change of code that I barely understand
<mattip> dunno really. You might have to call PyObject_GenericSetDict to resync the objects
<mattip> but maybe not, I am not sure
<ctismer> trying...
<ctismer> mattip: In the actual case, `PyObject_SetAttr(source, key, signalInstance)` is called. The initialization is `self->ob_dict = nullptr;`. Would I need to remove that initialization?
Julian has quit [Quit: leaving]
greedom has quit [Remote host closed the connection]
slav0nic has joined #pypy
Julian has joined #pypy
andythenorth has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<cfbolz> ctismer: I would try generic get dict first, it sounds like a small and mechanical change
<cfbolz> If it turns out to be a huge perf problem we can think again
<ctismer> cfbolz: just tried and got from 66 to 34 errors! Not ready, yet :)
<cfbolz> ctismer: you pessimist, that sounds great 😉
<LarstiQ> so that's 6 * 3 more weeks were we half the number of errors? ;)
<ctismer> cfbolz: I'm more than happy 😁
Julian has quit [Quit: leaving]
<ctismer> LarstiQ: down to 32 errors
Guest96 has joined #pypy
Corbin has quit [Ping timeout: 265 seconds]
Corbin has joined #pypy
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]