<headius>
master is green other than that exclude after fixing the size checks
_whitelogger has joined #jruby
<headius>
good morning!
<headius>
enebo: I modified that failing test to use RubyBoolean.hashCode for its reflection stuff, since that will always be available
<headius>
and always open to us
<headius>
enebo: it passed... I
<headius>
I'm going to move it out of expected fail
<enebo[m]>
ok well that's neat
<enebo[m]>
the expected fail was definitely why we never noticed complete was broken
<headius>
I am wondering if it started failing in the first place because the complete build was broken
<headius>
in any case we should have it up again now
<enebo[m]>
it has failed immediately from not finding org.jruby.Main for over a year
<headius>
enebo: yeah so there you go
<headius>
it broke when jruby-base was created and we weren't sure why and mkristian never jumped in to fix it
<headius>
we need to be able to take this all over so I think after 9.3 forks we should start guttin git
<headius>
guttin' git
<enebo[m]>
this also ended up uncovering that jruby-core never was a usable artifact although I suspect that has been true for years
<enebo[m]>
truth is all embedders just use jruby-complete
<headius>
jruby-core in 9.2 seems correct
<enebo[m]>
That was what I pasted last night
<enebo[m]>
It has no shaded stuff in it at all
<enebo[m]>
That is why it is 5M smaller than the jruby.jar in bin dist
<headius>
oh I see
<headius>
that's why the test expected it to be 10M but the lib/jruby.jar was also 15-16M
<headius>
I compared 9.2 lib/jruby.jar to your fixed jruby-core and they are pretty much the same
<headius>
but jnr-ffi is in 9.2 jruby-core for whatever reason
<headius>
so should we fix 9.2?
<enebo[m]>
yeah we should definitely add the missing stuff
<enebo[m]>
I really think if people were using this as an artifact they would have reported this
<enebo[m]>
although we do not have a jruby-base for 9.2 so if they did work around this issue or really desire jruby-base which is not too far off of what jruby-core was then that is the only risk in fixing core
<enebo[m]>
Actually maybe we should just leave it on 9.2. If someone reports it we can fix it
<headius>
yeah I agree
<headius>
leave it
<headius>
nobody has ever noticed because if they're using us from Java they just go straight to complete
<enebo[m]>
oh noes
<headius>
people did report wanting base though
<enebo[m]>
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy javadocs and sources from jruby-core) on project jruby-complete: Unable to find artifact.: Could not find artifact org.jruby:jruby-core:jar:sources:9.3.0.0-SNAPSHOT in sonatype (https://oss.sonatype.org/content/repositories/snapshots/)
<headius>
hardy har
<enebo[m]>
So my better fix did not actually fix this
<enebo[m]>
My original solution was to make a source directory tree and I put A.java there but I could just put a META-INF/something
<enebo[m]>
(copy javadocs and sources from jruby-core) @ jruby-complete ---
<enebo[m]>
so it skips source perhaps it just needs to skip docs
<enebo[m]>
theer are no docs in jruby-complete either
<enebo[m]>
adding some of our LICENSING files into a src tree for shaded
<enebo[m]>
so it will generate an empty source file which has semi-relevant files
<headius>
I will look into the sonatype rules
<enebo[m]>
yeah I am not sure if this is their's or just some rando maven thing
<headius>
If, for some reason (for example, license issue or it's a Scala project), you can not provide -sources.jar or -javadoc.jar , please make fake -sources.jar or -javadoc.jar with simple README inside to pass the checking.
<headius>
Scala? ðĪ
<enebo[m]>
hahah sbt perhaps does something different
<headius>
pretty weird statement
<enebo[m]>
scala is not code but merely type definitions
<headius>
so they have this as a nagging requirement but it's not particularly strict
<enebo[m]>
ok well what I am doing should work for source at least
<enebo[m]>
I am trying it now...I may hit a snag on docs but I thought I ran through with this fix before and it completed
<enebo[m]>
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy javadocs and sources from jruby-core) on project jruby-complete: Unable to find artifact.: Could not find artifact org.jruby:jruby-core:jar:javadoc:9.3.0.0-SNAPSHOT in sonatype (https://oss.sonatype.org/content/repositories/snapshots/)
<enebo[m]>
I admit this would still be wrong but I am just trying to understand this
<enebo[m]>
oh
<headius>
there's a second one configured for other profiles
<headius>
that uses core
<enebo[m]>
I was actually confused about that
<headius>
the way we intended to publish this was that the published jruby-core jar would be org.jruby.core and the one in lib would be org.jruby.dist
<headius>
but I think at that time we thought or knew that the core jar was not quite right for lib/jruby.jar as is
<headius>
that may have been because shading broke
<headius>
they could possibly be identical now but there's no way for us to detect which one lib/jruby.jar is
<headius>
bleh maybe we should stop publishing core
<headius>
that would eliminate a bunch of problems here
<enebo[m]>
ok so my change made the shading for core be to write to target but then copy out and for dist perhaps we need to explicitly specify lib/jruby.jar?
<headius>
so you did not break a normal build
<headius>
but I last ran this with -Pjruby_complete_jar_extended
<headius>
which uses the other shade
<headius>
release is also in here though so that would be the wrong name when you release it
<enebo[m]>
so this is a toggle?
<headius>
well when you select one of those profiles wrapping the second shade it changes the config
<headius>
and that config uses org.jruby.core
<headius>
so when you go to release with -Prelease it would have used the wrong one
<headius>
hmm
<headius>
I am leaning toward just making both of them org.jruby.dist
<headius>
we know the old core didn't work even if it had a different module name
<headius>
and base is available now
<headius>
so we will publish the same identical jar for jruby-core as we have in lib/jruby.jar and they will both be module org.jruby.dist
<headius>
that would be simplest
<headius>
most module users will want to use base anyway since you're not supposed to shade in modules
<headius>
and we will use that eventually
<enebo[m]>
seems ok to me
<enebo[m]>
The fake source fixed the issue in deployment (well it will run and push to sonatype
<enebo[m]>
I will see if it can close now
<headius>
the beginning of the end for jruby-core
<headius>
shading is a dying hack anyway
<headius>
that does fix this
<enebo[m]>
I learned something interesting about sonatype staging
<enebo[m]>
if you release 9.2.20.0 and 9.3.0.0 during the same session it puts that content into the same session/folders
<headius>
ah yeah
<headius>
so you would need to wipe it out before releasing
<enebo[m]>
I guess so that you can release those artifacts together atomicly?
<headius>
yeah
<enebo[m]>
That is pretty surprising to me
<headius>
that's how I do jnr releases all at once
<headius>
they are in a single staging repo and the whole thing gets released atomically
<headius>
it would take a million years to release them all if I didn't have that
<enebo[m]>
It makes sense now that I noticed it but I guess a lesson is to always remember to view contents before you close
<enebo[m]>
well release
<enebo[m]>
close you got another chance
<headius>
yeah I have gotten in that habit with jnr because once I had mismatched versions and one of the artifacts went into a different repo
<headius>
which clued me in that I forgot to update a version
<headius>
if I do it right they should all be together
<headius>
I pushed that change
<enebo[m]>
but I have two different versions of the same thing in the same repo
<headius>
in the future jruby-base as a module will go under lib/modules and the launchers can detect that
<headius>
we just can't detect which module lib/jruby.jar is without opening it
<headius>
I don't want to say modules were a mistake but after all that work they still seem pretty half baked
<enebo[m]>
I keep thinking their main benefit will be knowing about total usage in a way to do better runtime analysis but I have not seen anything about them actually doing that.
<enebo[m]>
Coupling that with sealed classes to restrict it even more
<headius>
varargs bug seems to only be primitives
<enebo[m]>
ok it closed so sonatype is happy
<headius>
json warnings are due to new openstruct adding aliases for all of Object's methods, which means it tries to alias stuff like `instance_eval` to `instance_eval!` and we warn about that
<headius>
I can work around it by making these goofy names be forced frame but also filed a bug for them to revert this behavior: https://github.com/ruby/ostruct/issues/30
lopex has joined #jruby
<headius>
enebo: we can punt on this by rolling back to 0.1.0 which is what 2.6.8 ships
<headius>
ostruct 0.1.0
<headius>
I don't want to hack around it without making a stink
<headius>
hmm do I mark this fixed, or just revert ostruct and punt the issue to 9.3.1
<headius>
the warnings are still out there if you upgrade ostruct
<headius>
that one is done
<enebo[m]>
lunch...doing some testing and I glanced at the window-31j issue. MRI 2.6.8 also passes this. It may be a bit more involved to understand what is happening
<headius>
9.2.19 working as well means something's regressed here
<headius>
so I think we need to fix it
<headius>
I will leave that to you for now and look into the varargs bug
<headius>
the varargs issue also works in 19
<headius>
ahh this was due to an optimization byteit101 did for populating a primitive arg array I think
<byteit101[m]>
I didn't optimize primitive arrays, just object arrays, IIRC
<headius>
yeah it is just broken
<headius>
if (varArrayType.isPrimitive()) {
<headius>
varArrayType for a primitive int array is [I
<headius>
it should be component type
<headius>
I missed it too
<headius>
apparently we have no tests for calling primitive varargs ðĪŠ
<headius>
honestly how does that get this far without being caught
<byteit101[m]>
I remember I didn't want to figure out what to test for the primitive stuff, so just stuck that if there. Oops :-)
<headius>
must not be many primitive varargs arrays out there
<headius>
not sure I ever would have caught that just reading the code
<headius>
trivial fix anyway and stepping found it right away
<byteit101[m]>
So I correct my earlier statement: I didn't think I optimized primitive arrays
<headius>
you didn't
<headius>
you shortcutted object arrays since they don't have to use reflection
<headius>
but because of this primitive arrays also did it
<headius>
optimizing primitive arrays would require calling out to something that could cast to e.g. int[] and set directly (which might be worth it, except apparently this is really uncommon in varargs!)
<headius>
zero hits for any primitive... in JI specs
<enebo[m]>
JRuby always worked from the premise we make an encoding for a bytelist and add data as we build up the string
<enebo[m]>
MRI passes in the encoding for the lex and another enc variable for whatever the string will be
<enebo[m]>
It starts out as null
<enebo[m]>
So when it enters the unicode escape processing it still has no encoding at all
<enebo[m]>
but it sees it is processing unicode and sets it to utf-8
<enebo[m]>
in the case it does not see a condition like that it will stay null
<enebo[m]>
at the end of the method it will say "what no encoding yet...let's just use the other encoding we passed in"
<enebo[m]>
So the problem I slowly backed into was trying to assume a default encoding up front but that means we end up losing this state to realize we have not figured out the strings encoding yet
<headius>
so is it not supposed to be utf-8 when first parsed?
<enebo[m]>
This worked for many years too but in fixing the last known parsing errors I finally realize I am at a dead end
<headius>
I don't think I'm following... if the bytes are valid Windows-31J why does it fail?
<headius>
or are we doing something with the bytes that makes them unicode only?
<enebo[m]>
It reads a file which has unicode text escapes and quotes
<enebo[m]>
that is all ascii
<enebo[m]>
it then evals that ascii string which makes a utf-8 string
<enebo[m]>
but because we set encoding to windows-31J it assumes the string will starts as a windows-31J string and not a null one until we determine what it should be
<enebo[m]>
So I will need to change strterm to mimic this which means pushing through an extra nullable encoding field OR some state saying we have not figured that out yet
<enebo[m]>
In both cases they end up as out params
<enebo[m]>
which I suspect is why I did it by leaning on bytelist
<basshelal[m]>
Sorry if you're busy but see this in case you haven't
<basshelal[m]>
The guys at Jenkins use JNR-FFI, that's freaking awesome and freaking terrifying ðð
<headius>
basshelal: thanks, had not seen that
<headius>
doesn't sound critical but I know maven projects get cranky if you have differing versions
<headius>
enebo: ohhhh
<headius>
if it's all ascii despite the escapes why is it not being encoded as US-ASCII at that point?
<headius>
I'm still not clear why it isn't able to be forced to W31J if the bytes are valid as W31J
<enebo[m]>
well it is likely whatever is the default external encoding at that point
<enebo[m]>
the reason is when I make strings in a window31J file I just initially set the encoding to that
<headius>
ohhhh
<enebo[m]>
Which is normally what it is
<headius>
so it gets marked as W31J but it has unicode escapes in it
<enebo[m]>
but in the case of an explicit unicode escape it must be utf-8 then
<headius>
and we don't know how to handle unicode escapes in W31J string
<enebo[m]>
well at the point of the error we don't know that we did not have a reason for it to be windows-31j or whether it is fresh
<enebo[m]>
If there were already valid windows-31J we would have set it to that already then run into a rnado unicode escape and in that case we should error
<enebo[m]>
in this case there was nothing but unicode escapes so it should have not errored and become a utf-8 string
<enebo[m]>
So the good news is I completely understand why it is broken
<enebo[m]>
I just dislike the fix I will have to make
<headius>
oops, I need to revert ostruct tests too
<headius>
once again I am burned by committing to master
<enebo[m]>
yeah
<enebo[m]>
I am getting more into the groove but near release it is much more tempting
<headius>
yeah
<enebo[m]>
I do totally know the fix but I have to audit all paths to see which ones may decide to set the encoding