otisolsen70 has quit [Remote host closed the connection]
otisolsen70 has joined #pypy
danchr_ is now known as danchr
slav0nic has joined #pypy
<cfbolz>
mattip: should we mention in the release notes that this will be the last release of 3.7?
<mattip>
good point
<cfbolz>
mattip: I can add it
<cfbolz>
I am doing minor things in there anyway
<mattip>
cool
<mattip>
64a22a19910f is on py3.9, probably belongs on default
<cfbolz>
oh ouch, thanks
<cfbolz>
mattip: both done
<cfbolz>
mattip: if we release with the missing dict copy stuff (and it looks like we probably will) we should maybe mention that too in the announcement?
<fijal>
cfbolz: hi
<mattip>
ok
<mattip>
thanks for the fixes
<mgorny>
it seems that parallel (-j 12) compileall for pypy3.9 stdlib hangs for me (-j1 is fine)
<mgorny>
i wonder if it could mean that pypy3.9 has some generic concurrency issues
<cfbolz>
mgorny: interesting, what is the precise command line?
<mgorny>
our patch had .copy(), so i didn't notice the list()
<mgorny>
well, first hang was after Compiling '/tmp/portage/dev-python/pypy3-7.3.8_rc1/image/usr/lib/pypy3.9/zoneinfo/_common.py'...
<mgorny>
so prolly last file
<mgorny>
i've added more debug to check if it hangs after last -O file or before first -O file
<cfbolz>
ok, thank you
<cfbolz>
still no local hang
<mgorny>
er, definitely after last file, as i didn't remove -q from other runs
<mattip>
maybe some file/resource being help open?
<mgorny>
ok, happens without -O too
<mgorny>
(but more commonly with -O...)
<cfbolz>
I am getting it now too
<cfbolz>
progress
<mattip>
mgorny: the 3.9 binary is supposed to be called pypy3.9-c and the shared library libpypy3.9-c.so
<mattip>
so it can be put next to the other shared libraries
<cfbolz>
mgorny: when I strace it, it doesn't hang of course ;-)
<mgorny>
mattip: but i don't think renaming it matters here? i'm trying minimal changse from 3.8
<mgorny>
cfbolz: according to gdb, they're all waiting on some futex
<mgorny>
i can try rebuilding pypy3 with debug symbols if that helps
<mattip>
ok, just curious if the new naming feature is useful - but let's find this error first
<mgorny>
mattip: for end users, certainly. however, on gentoo i don't really have the resources to support more than one pypy3 version
<mattip>
+1
Atque has joined #pypy
<cfbolz>
mgorny: when it hangs, there are 12 subprocesses still open. that means the shutdown at the end is definitely not working right
<cfbolz>
anyway, need to leave for now
otisolsen70 has quit [Ping timeout: 256 seconds]
<mattip>
I think using lower case locale solves the failing own test/test_app_main.py tests:
<mattip>
sudo docker run -e LANG=en_us.utf-8 -e LC_ALL=en_us.utf-8 ...
<cfbolz>
mattip: "cool"
<mattip>
let's see next time a x86_64 own tests run
Atque has quit [Quit: ...]
<mgorny>
i've added initial pypy3.9 ebuilds to gentoo (masked)
<mgorny>
i've confirmed that the layout works for building pillow
<mgorny>
in the next hour, i'm going to start rebuilding everything
Atque has joined #pypy
greedom has joined #pypy
Julian has joined #pypy
greedom has quit [Remote host closed the connection]
greedom has joined #pypy
greedom has quit [Remote host closed the connection]
greedom has joined #pypy
Atque has quit [Quit: ...]
otisolsen70 has joined #pypy
otisolsen70 has quit [Remote host closed the connection]
otisolsen70 has joined #pypy
Julian has quit [Quit: leaving]
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Guest96 has joined #pypy
Guest96 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<mgorny>
cfbolz: bad news: turns out compileall also hangs on various other packages
<mgorny>
including very small packages :-(
<cfbolz>
mgorny: yeah, we think it's a more general problem with concurrent futures
<cfbolz>
So definitely a release blocker :-(
<mgorny>
cfbolz: do you have some suggestions how i could help? i suppose it's much easier to repro on my hardware
<mattip>
what disk are you running? I have a nvme ssd and cannot reproduce the hang
<mattip>
does it help if you do --jit off ?
<mgorny>
nvme as well, though the builds are happening on tmpfs
* tumbleweed
hasn't tried started trying to do the new layout with 3.8 yet, but if I get time I'll poke at it in the next days
<tumbleweed>
fwiw, make_ssl_data.py isn't valid Python3, that could be breaking your compileall
* tumbleweed
uses his own py_compile wrapper, without the benefit of compileall's paralellism
<mgorny>
tumbleweed: it happens on random py3 packages as well, e.g. tomli
<mgorny>
(and tomli is 4 .py files...)
<tumbleweed>
err I meant isn't valid python2.7, it's python3, so it breaks py_compile for pypy2.7
<tumbleweed>
right
<tumbleweed>
sounds like crappy concurrent futures
<mgorny>
actually, it might be even easier to reproduce on small packages, i guess
<cfbolz>
mgorny: yes, thanks for the hint with tomli
<mgorny>
cfbolz: actually, much harder to reproduce with it
<mgorny>
it seems that was one-time lucky streak ;-)
<mgorny>
mattip: --jit off doesn't help
<cfbolz>
seems trying to compileall multiprocessing hangs for me, and is a lot less files than the whole stdlib
<mgorny>
i managed with just one file ;-)
<mgorny>
but man, the logic is so complex :-(
<mgorny>
that said, i know that there were some serious multiprocessing bugs in cpython
<mgorny>
i had a tool (gemato) that i've made parallel at some point but i had to revert that because some users had weird hangs i couldn't ever reproduce
<mgorny>
and that made very little sense -- i suspect it was some bug in cpython that might still be there
<mattip>
does lsof help show which files are open?
<mattip>
mgorny: if we can't fix 3650, we should switch for the release
<mgorny>
and i'm going to try rebuilding all python packages in my test environment
<mattip>
the benefit of fork (lower memory requirements) comes at great cost (don't fork multithreaded processes)
<cfbolz>
just typing stuff and using irc as my rubber duck: it seems from gdb that some of the forked processes don't ever start doing any useful work, because they immediately deadlock blocking on some semaphore
<cfbolz>
and it seems that the lock that we deadlock on could be the sys.stdout lock??!
<mgorny>
maybe cpython uses different kind of underlying locks or sth?
<mattip>
maybe the debug printing?
<cfbolz>
mattip: possible
* cfbolz
switches to os.write
lazka has quit [Quit: Ping timeout (120 seconds)]
lazka has joined #pypy
<cfbolz>
mattip: thanks, that makes sense, no it deadlocks somewhere else
<mattip>
os.scandir left the fd in a bad state, I don't know whether that is part of the compile_all code?
<cfbolz>
mattip: not totally impossible, but unlikely I think