<antocuni>
ronan: I've tried to compile pypy in the hpy-0.0.2 branch. When running extra_tests/hpy_tests I get random segfaults/aborts/bus errors etc. so I think there is some memory corruption going on
<antocuni>
and the applevel tests all pass
<antocuni>
are you aware of it? Did you investigate already and/or have clues?
<Hodgestar>
antocuni: Re HPyUnicodeBuilder_XXX: Perhaps instead of having many HPyUnicode builders, the HPyBytesBuilder can have a "asStr" or "asUnicode" method that takes one of ASCII, UTF8, UCS{1,2,4} as the encoding and returns a handle to a Python unicode/str object?
<antocuni>
Hodgestar: I'm not sure to understand what you mean. Could you please provide a code example?
<ronan>
antocuni: no, I wasn't aware of the issue
<antocuni>
I'm trying to bisect to find when the bug was introduced, but it looks like extra_tests have not passed for a while
<antocuni>
do you confirm you also had to fix it/comment it out?
<ronan>
antocuni: yes, I have that issue if I use pytest 2.9.2, but not with a more recent version
<antocuni>
ah ok. I always use the pytest version which is bundled with pypy, didn't know we can also use a more recent one
<antocuni>
so I confirm that the nightly build passes, but my -O2 custom build aborts
* antocuni
tries to build with -Ojit
<Hodgestar>
antocuni: I can try write a code example. My bigger picture idea though is to avoid having a complex HPyUnicodeBuilder with different builders for different character layouts and to rather use the HPyBytesBuilder and then have a function that hands that buffer to the Python implementation behind the scenes and lets it do the best it can with the format. I think in practice it's not that different to the HPyUnicodeBuilder_XXX, but I'm trying to not
<Hodgestar>
have the API suggest that HPyUnicodeBuilder_UCS4 can be expected to behave as nicely as HPyUnicodeBuilder_UTF8, or that there is really equal support for all five character encoding formats.
<antocuni>
yes, but I'm trying to imagine how such an API would look like
<antocuni>
e.g., we can call HPyUnicodeBuilder_UCS4(size, &buffer), where &buffer is a HPy_UCS4**
<antocuni>
but if we don't distinguish between kinds, what is the type of the buffer?
<antocuni>
And similarly for "size": it represent the number of characters, but the size in bytes depends on the encoding
<antocuni>
so, it seems that we get the RPython abort only with -O2. PyPys translated with -Ojit can run extra_tests smoothly
<antocuni>
even with --jit off
<Hodgestar>
I'm wondering if it is possible to write a streaming encoder that holds only a few bytes of extra storage and converts to whatever encoding the underlying Python likes most?
ronan has quit [Ping timeout: 268 seconds]
<antocuni>
I wonder how serious is the bug. Is it a real undefined behavior which doesn't manifest with -Ojit just by chance? Or maybe it's a smaller issue which manifests only with the set of options enabled by -O2 and thus will never affect JIT builds?
ronan has joined #hpy
<antocuni>
Hodgestar: what is a streaming encoder?
<Hodgestar>
antocuni: One feeds in N bytes at a time and the encoder spits out M bytes of output. It's complicated though because we're trying to avoid slow copies. :/
<antocuni>
maybe, but do you have any concrete use case in which it would be useful?
ronan has quit [Ping timeout: 244 seconds]
ronan has joined #hpy
<antocuni>
I'm confused. I seemed to remember that -O2 and -Ojit builds differed in more that just the JTT (i.e. the enabled/disabled a slightly different set of objspace options), but it seems that nowadays they are the very same build
<antocuni>
cfbolz: do you confirm? ^^^
<cfbolz>
antocuni: yeah, we tried to unify that a lot
<cfbolz>
Since quite a while actually
<antocuni>
so basically, running pypy --jit off is essentially the same as running a -O2 pypy?
<cfbolz>
antocuni: 'almost'
<cfbolz>
But yes, quite close
<antocuni>
then I have no idea why this bug manifests in a very reproducible way with -O2 but never with -Ojit :(
<antocuni>
ronan: so, assuming that we fix this -O2 but with extra_tests, what is left to do before being able to release hpy-0.0.2?
<ronan>
I think it's ready now
<antocuni>
cool
<antocuni>
but I think I'd like to investigate a bit this problem before releasing
<antocuni>
we also need to make sure that our local copy of hpy is in sync with the git branch release/0.0.2
<antocuni>
fangerer: what is the GraalPython status w.r.t 0.0.2?
<antocuni>
if I comment out test_foo, the test consistently pass. If I include test_foo, I consistently get this error
<antocuni>
uhm wait, this is a FatalError which is raised by the test itself
<antocuni>
ok, I think that what happens is that handles are mapped to "random" W_Objects (or maybe even to random addresses in memory), so all kind of confusion occurs