09:34 <arigato> vext01: this warning means: "I'm not finding the directory lib-python/3/ or lib_pypy/ when I look in the place where the executable is or any parent directory"

10:14 <vext01> arigato: i see

10:20 <vext01> arigato: those dirs are in ../../

10:20 <vext01> should be ok, right?

10:20 <vext01> the pypy executable is a symlink from the usession dir, in case that matters

10:23 <vext01> yes, it's to do with the symlink

10:23 <vext01> is the warning incorrect then?

10:30 <cfbolz> vext01: hm, where is the *actual* file?

10:30 <cfbolz> in goal/

10:30 <cfbolz> or in usession*/testing_1 ?

10:35 <vext01> hi cfbolz

10:35 <vext01> i copied it into pypy/goal

10:35 <vext01> and it works

10:35 <vext01> but if you symlink it into tha tsame place, it does not

10:36 <vext01> (i got the idea to make a symlink from the warning itself)

10:37 <cfbolz> the warning says to do the opposite though: the binary must always be in goal/. but it's fine to put a symlink to it somewhere else

10:40 <vext01> ohhh

11:00 <vext01> ok, now build cffi imports

11:00 <vext01> https://gist.github.com/vext01/2cf6c6ac73881685aba763ade6fae19c

11:00 <vext01> i've put prints in to see where it thinks the lib_pypy dir is

11:01 <vext01> it adds the top-level src dir as a PYTHONPATH

11:01 <vext01> is that right?

11:08 <vext01> ah, i think i see it

11:13 <vext01> nope...

11:13 <vext01> any ideas?

11:37 <cfbolz> vext01: hm, can you try to see whether afterwards "import hashlib" still warns?

11:47 <vext01> cfbolz: yes, the same error

11:48 <cfbolz> vext01: that probably means one of the required libs isn't installed?

11:48 <vext01> ok, i'll check that next

11:48 <vext01> (should be right though, we always had ssl support in the package)

11:49 <cfbolz> I really don't know how that works unfortunately

11:49 <cfbolz> vext01: iirc blake2b is special though

11:54 <vext01> this could be a libressl problem

11:55 <vext01> later in the trace:

11:55 <vext01> ValueError: unsupported hash type blake2b

11:56 <vext01> is blake2b considered weak?

11:56 <vext01> it's fairly new and looks strong at a glance.. hrm

11:56 <cfbolz> vext01: no, but it's not part of ssl I think

11:57 <cfbolz> vext01: look at lib_pypy/_blake2

11:57 <vext01> yeah, but openbsd has a fork of openssl

12:02 <cfbolz> vext01: it looks to me like lib_pypy/_blake2 just has *all* the c code for blake2

12:03 <cfbolz> so you'll have to debug a bit why that doesn't work

12:03 <cfbolz> does "import _blake2" work, for example?

12:08 <vext01> i've been speaking to one of the libressl devs

12:08 <vext01> it doesnt support that algorithm

12:08 <vext01> he's gonna patch it out for me, since he's more experienced

12:34 <LarstiQ> what support does libressl need to bring if lib_pypy/_blake2 has all of the code?

12:38 <cfbolz> vext01: stop, this has really nothing to do with the ssl lib

12:38 <cfbolz> vext01: pypy ships all the C code it needs for blake2

12:38 <cfbolz> so you need to find out why that doesn't build

13:28 <vext01> oh right

17:34 <Guest12> I have a gunicorn/flask app that runs under pypy and creates a lot of objects that end up being effectively read only.  I am trying to make use of the gunicorn prefork model and copy on write memory to only keep one copy of the read only objects.  To do this I create all the objects in the master process pre fork and run a gc.collect().  This

17:34 <Guest12> works initially and memory is shared as best I can tell but as soon as GC runs in the forked process all the memory looks like it gets copied. My understanding was once an object is copied out of the nursery it should not be touched by GC anymore since  pypy is not updating references in each object.  Is my understanding off here or is there

17:34 <Guest12> something else going on that I missed?

17:49 <Guest12> fwiw this seems to work in cpython using gc.freeze() but this is not implemented in pypy

18:11 <Dejan> Guest12, which version?

18:12 <Guest12> Python 3.8.16 PyPy 7.3.11

18:16 <arigato> Dejan: yes, there are a few bits in the header of all objects, and a full GC pass requires marking and unmarking these bits

18:16 <Guest12> even for old generations?

18:16 <arigato> there is a theory about how to solve it, but we never got around to implement it

18:16 <arigato> yes

18:17 <Dejan> arigato, i assume the message was aimed at Guest12

18:17 <Dejan> :)

18:17 <arigato> a full GC pass requires marking all live objects, and then freeing the rest

18:17 <arigato> Dejan: oops :-) of course

18:17 <Dejan> full GC pass is quite expensive

18:19 <Guest12> is the "theory about how to solve it" written down somewhere?

18:19 <Guest12> would be curious to see that

18:19 <arigato> Dejan: yes, but it occurs rarely and is actually run incrementally so it doesn't create too long pauses, so it's usually fine (and better than refcounting for performance). This problem with forks is a drawback

18:21 <arigato> Guest12: I can't tell you exactly, cfbolz might have a reference. The idea is to keep the mark bits outside the object headers, in a few temporary pages of memory, and have a way to map object addresses to the correct bit

18:22 <Guest12> does a minor collection touch everything in the nursery?

18:22 <cfbolz> Yes

18:23 <arigato> a minor collection touches all objects in the nursery that are still alive. it ignores completely the memory used by objects that are dead

18:23 <arigato> (but the nursery is tiny (a few MBs) so it's not relevant for this fork issue)

18:23 <Guest12> got it

18:23 <Guest12> Thanks for confirming

18:25 <arigato> also, I'm not sure that gc.freeze() on CPython fixes everything, either: as soon as you walk through the objects, then reference counters will be constantly updated in CPython, thus un-sharing potentially everything

18:25 <arigato> it works as long as you don't look at most objects

18:26 <Guest12> from my initial testing it seemed to keep memory stable, I could very well have had a bad test though

18:27 <arigato> without gc.freeze() then CPython will unshare everything at the next GC last-level collection (which is rarer than in PyPy I guess); with gc.freeze() then that problem is avoided, but not the refcount one

18:28 <arigato> but it depends more on how your program accesses the read-only objects

18:29 <Guest12> interesting, I will have to go back and verify my test.  It might just be the change was not as noticeable compared to pypy since it only touched some objects instead of all of them

18:29 <arigato> maybe.

18:30 <Guest12> The calls I make in the test would have likely included a GC run which would impact PYPY but not CPython

18:31 <arigato> yes, if you test for only a short time, then likely it includes a GC run but not the long-term behaviour

18:32 <Guest12> makes sense