11:13 <mgorny> mattip, cfbolz: redis(-py) seems to be another package affected by the spawn problem -- here we can't pickle _thread.Lock being part of the client

11:14 <mgorny> i suppose the only solution is to have all consumers using redis + multiprocessing force "fork" method, correct?

11:15 <mgorny> though i suppose passing threading.Lock() to a forked process is kinda weird

11:16 <mgorny> maybe they should switch to multiprocessing.Lock instead?

11:16 <cfbolz> mgorny: and they simply don't work on mac, I assume?

11:16 <cfbolz> I don't know, the real solution would be to find this bug :-(

11:17 <cfbolz> but I am a quite stuck with that at the moment

11:18 <mgorny> cfbolz: probably not, i find it hard to test since it relies on working redis instance and their test suite relies on docker

11:18 <cfbolz> right

11:24 <mgorny> after resolving locks, now lambdas are a problem

11:25 <mgorny> basically a lot of work to make redis client class serializable, and i'm not even sure if they should really be doing that

11:26 <cfbolz> mgorny: basically the fork default lets projects be a little bit lazy. because some unpicklable things will be around in the child process from the fork

11:33 <mattip> redis-py does support macOS, but maybe that specific feature is disabled?

11:35 <mattip> there is this stale and closed issue https://github.com/redis/redis-py/issues/606

11:35 <mgorny> mattip: i don't grep darwin|macos|osx

11:35 <mgorny> maybe they're just ignoring the test failures

11:39 <mattip> they are not running macOS tests

11:40 <mattip> but the issue I pointed to was from macOS

11:40 <mattip> I think unilaterally turning on spawn instead of fork is problematic

11:41 <mgorny> yes

11:41 <mattip> if we leave fork as the default, does the hang show up in places outside compileall?

11:41 <mgorny> but the deadlock is also bad :-(

11:42 <mgorny> i mean, i can just switch gentoo to use serial compileall on pypy3.9 but i'm afraid this could also affect other multiprocessing uses

11:43 <cfbolz> particularly since compileall is a relatively "simple" use of multiprocessing

11:43 <cfbolz> ie it does not use threads itself

12:16 <mgorny> i've filed https://github.com/redis/redis-py/issues/1931 with somewhat detailed explanation

12:47 <mattip> mgorny: thanks

16:57 <cfbolz> hmmmm

16:57 <cfbolz> I wonder what happens if I revert the multiprocessing code to that of the 3.8 stdlib

17:04 <cfbolz> (I mean I don't want to ship that. but is the deadlock still there)

18:19 <cfbolz> ok, I tried to basically bisect CPython commits too see which ones hang

18:20 <cfbolz> and it points to this: https://github.com/python/cpython/commit/1ac6e379297cc1cf8acf6c1b011fccc7b3da2cbe

18:25 <cfbolz> unfortunately, when I back that commit out, it still deadlocks :-(

18:34 <mattip> did anything change in _multiprocessing?

18:37 <cfbolz> good question

18:37 <cfbolz> anyway, if I go back to *before* that commit, the deadlock is definitely gone

19:44 <mattip> ahh, cool

19:45 <mattip> so it is something after that commit

19:46 <cfbolz> yeah, maybe