<headius> master is green other than that exclude after fixing the size checks
_whitelogger has joined #jruby
<headius> good morning!
<headius> enebo: I modified that failing test to use RubyBoolean.hashCode for its reflection stuff, since that will always be available
<headius> and always open to us
<headius> enebo: it passed... I
<headius> I'm going to move it out of expected fail
<enebo[m]> ok well that's neat
<enebo[m]> the expected fail was definitely why we never noticed complete was broken
<headius> I am wondering if it started failing in the first place because the complete build was broken
<headius> in any case we should have it up again now
<enebo[m]> it has failed immediately from not finding org.jruby.Main for over a year
<headius> enebo: yeah so there you go
<headius> it broke when jruby-base was created and we weren't sure why and mkristian never jumped in to fix it
<headius> we need to be able to take this all over so I think after 9.3 forks we should start guttin git
<headius> guttin' git
<enebo[m]> this also ended up uncovering that jruby-core never was a usable artifact although I suspect that has been true for years
<enebo[m]> truth is all embedders just use jruby-complete
<headius> jruby-core in 9.2 seems correct
<enebo[m]> That was what I pasted last night
<enebo[m]> It has no shaded stuff in it at all
<enebo[m]> That is why it is 5M smaller than the jruby.jar in bin dist
<headius> oh I see
<headius> that's why the test expected it to be 10M but the lib/jruby.jar was also 15-16M
<headius> I compared 9.2 lib/jruby.jar to your fixed jruby-core and they are pretty much the same
<headius> but jnr-ffi is in 9.2 jruby-core for whatever reason
<headius> so should we fix 9.2?
<enebo[m]> yeah we should definitely add the missing stuff
<enebo[m]> I really think if people were using this as an artifact they would have reported this
<enebo[m]> although we do not have a jruby-base for 9.2 so if they did work around this issue or really desire jruby-base which is not too far off of what jruby-core was then that is the only risk in fixing core
<enebo[m]> Actually maybe we should just leave it on 9.2. If someone reports it we can fix it
<headius> yeah I agree
<headius> leave it
<headius> nobody has ever noticed because if they're using us from Java they just go straight to complete
<enebo[m]> oh noes
<headius> people did report wanting base though
<enebo[m]> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy javadocs and sources from jruby-core) on project jruby-complete: Unable to find artifact.: Could not find artifact org.jruby:jruby-core:jar:sources: in sonatype (https://oss.sonatype.org/content/repositories/snapshots/)
<headius> hardy har
<enebo[m]> So my better fix did not actually fix this
<enebo[m]> My original solution was to make a source directory tree and I put A.java there but I could just put a META-INF/something
<enebo[m]> (copy javadocs and sources from jruby-core) @ jruby-complete ---
<enebo[m]> so it skips source perhaps it just needs to skip docs
<enebo[m]> theer are no docs in jruby-complete either
<enebo[m]> adding some of our LICENSING files into a src tree for shaded
<enebo[m]> so it will generate an empty source file which has semi-relevant files
<headius> I will look into the sonatype rules
<enebo[m]> yeah I am not sure if this is their's or just some rando maven thing
<headius> If, for some reason (for example, license issue or it's a Scala project), you can not provide -sources.jar or -javadoc.jar , please make fake -sources.jar or -javadoc.jar with simple README inside to pass the checking.
<headius> Scala? ðŸĪ”
<enebo[m]> hahah sbt perhaps does something different
<headius> pretty weird statement
<enebo[m]> scala is not code but merely type definitions
<headius> so they have this as a nagging requirement but it's not particularly strict
<enebo[m]> ok well what I am doing should work for source at least
<enebo[m]> I am trying it now...I may hit a snag on docs but I thought I ran through with this fix before and it completed
<enebo[m]> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:copy (copy javadocs and sources from jruby-core) on project jruby-complete: Unable to find artifact.: Could not find artifact org.jruby:jruby-core:jar:javadoc: in sonatype (https://oss.sonatype.org/content/repositories/snapshots/)
<enebo[m]> LOL
<enebo[m]> [INFO] --- maven-javadoc-plugin:2.7:jar (attach-javadocs) @ jruby-core ---
<enebo[m]> [INFO]
<enebo[m]> not even a message
<headius> sorry got distracted for a bit
<headius> so past source but still stuck on javadocs
<enebo[m]> ok first 9.3 bug! https://github.com/jruby/jruby/issues/6832
<enebo[m]> I am doing something to work around this
<enebo[m]> Or trying
<enebo[m]> shaded/src/main/java/Nothing.java
<enebo[m]> with a comment saying this is just for sonatype to generate a javadocs jar
<headius> woot
<headius> don't put it in default package
<headius> Java module system rejects modules with classes in default package
<headius> this person opened three issues today
<headius> I have no idea why this encoding one should pass
<headius> it is utf-8 content forced into windows-31j and then expected to parse
<enebo[m]> this is not used as source
<enebo[m]> I mean it will end up in jruby-complete-source.jar and the javadocs but nothing else
<headius> ok, as long as it does not end up in the published jar
<enebo[m]> The code he uses is ascii strings which are win32j butthey are unicode escapes
<enebo[m]> Maybe MRI is a little different codewise there
<headius> pretty weird
<enebo[m]> headius: not in the jar people use for reals just the javadoc and sources jars
<headius> I will look into these bugs
<enebo[m]> else if (codepoint >= 0x80) {
<enebo[m]> if (*encp && utf8 != *encp) {
<enebo[m]> rb_encoding *utf8 = rb_utf8_encoding();
<headius> enebo: problem with your jruby-core copying
<headius> lib/jruby.jar is supposed to be org.jruby.dist module
<headius> if we want it to be exactly the same as jruby-core we can do that but we have org.jruby.dist in the bash and native launchers
<headius> or we need to repackage from core to lib
<headius> or we rename the core module to dist
<enebo[m]> I am a bit confused now...I just changed outputDir from lib/jruby.jar in shaded to be default and then copy from there.
<enebo[m]> I don't see how the contents itself of lib/jruby.jar would be any different than before
<headius> right, but the way it was building before it got shaded into lib using the org.jruby.dist module in manifest
<headius> that might be what broke jruby-core shading
<headius> I think we could use a second jar plugin execution to rejar with the modified manifest
<enebo[m]> but wouldn't I just have put the wrong module into jruby-core and lib/jruby would still be the same as before?
<headius> hmm
<headius> I do see it in the shade config
<enebo[m]> a78b52ee1310ac655927689de966fad2e7c5808b
<headius> oh there's two shades?
<enebo[m]> I admit this would still be wrong but I am just trying to understand this
<enebo[m]> oh
<headius> there's a second one configured for other profiles
<headius> that uses core
<enebo[m]> I was actually confused about that
<headius> the way we intended to publish this was that the published jruby-core jar would be org.jruby.core and the one in lib would be org.jruby.dist
<headius> but I think at that time we thought or knew that the core jar was not quite right for lib/jruby.jar as is
<headius> that may have been because shading broke
<headius> they could possibly be identical now but there's no way for us to detect which one lib/jruby.jar is
<headius> bleh maybe we should stop publishing core
<headius> that would eliminate a bunch of problems here
<enebo[m]> ok so my change made the shading for core be to write to target but then copy out and for dist perhaps we need to explicitly specify lib/jruby.jar?
<headius> so you did not break a normal build
<headius> but I last ran this with -Pjruby_complete_jar_extended
<headius> which uses the other shade
<headius> release is also in here though so that would be the wrong name when you release it
<enebo[m]> so this is a toggle?
<headius> well when you select one of those profiles wrapping the second shade it changes the config
<headius> and that config uses org.jruby.core
<headius> so when you go to release with -Prelease it would have used the wrong one
<headius> hmm
<headius> I am leaning toward just making both of them org.jruby.dist
<headius> we know the old core didn't work even if it had a different module name
<headius> and base is available now
<headius> so we will publish the same identical jar for jruby-core as we have in lib/jruby.jar and they will both be module org.jruby.dist
<headius> that would be simplest
<headius> most module users will want to use base anyway since you're not supposed to shade in modules
<headius> and we will use that eventually
<enebo[m]> seems ok to me
<enebo[m]> The fake source fixed the issue in deployment (well it will run and push to sonatype
<enebo[m]> I will see if it can close now
<headius> the beginning of the end for jruby-core
<headius> shading is a dying hack anyway
<headius> that does fix this
<enebo[m]> I learned something interesting about sonatype staging
<enebo[m]> if you release and during the same session it puts that content into the same session/folders
<headius> ah yeah
<headius> so you would need to wipe it out before releasing
<enebo[m]> I guess so that you can release those artifacts together atomicly?
<headius> yeah
<enebo[m]> That is pretty surprising to me
<headius> that's how I do jnr releases all at once
<headius> they are in a single staging repo and the whole thing gets released atomically
<headius> it would take a million years to release them all if I didn't have that
<enebo[m]> It makes sense now that I noticed it but I guess a lesson is to always remember to view contents before you close
<enebo[m]> well release
<enebo[m]> close you got another chance
<headius> yeah I have gotten in that habit with jnr because once I had mismatched versions and one of the artifacts went into a different repo
<headius> which clued me in that I forgot to update a version
<headius> if I do it right they should all be together
<headius> I pushed that change
<enebo[m]> but I have two different versions of the same thing in the same repo
<headius> in the future jruby-base as a module will go under lib/modules and the launchers can detect that
<headius> we just can't detect which module lib/jruby.jar is without opening it
<headius> I don't want to say modules were a mistake but after all that work they still seem pretty half baked
<enebo[m]> I keep thinking their main benefit will be knowing about total usage in a way to do better runtime analysis but I have not seen anything about them actually doing that.
<enebo[m]> Coupling that with sealed classes to restrict it even more
<headius> varargs bug seems to only be primitives
<enebo[m]> ok it closed so sonatype is happy
<headius> json warnings are due to new openstruct adding aliases for all of Object's methods, which means it tries to alias stuff like `instance_eval` to `instance_eval!` and we warn about that
<headius> I can work around it by making these goofy names be forced frame but also filed a bug for them to revert this behavior: https://github.com/ruby/ostruct/issues/30
lopex has joined #jruby
<headius> enebo: we can punt on this by rolling back to 0.1.0 which is what 2.6.8 ships
<headius> ostruct 0.1.0
<headius> I don't want to hack around it without making a stink
<headius> hmm do I mark this fixed, or just revert ostruct and punt the issue to 9.3.1
<headius> the warnings are still out there if you upgrade ostruct
<headius> that one is done
<enebo[m]> lunch...doing some testing and I glanced at the window-31j issue. MRI 2.6.8 also passes this. It may be a bit more involved to understand what is happening
<headius> 9.2.19 working as well means something's regressed here
<headius> so I think we need to fix it
<headius> I will leave that to you for now and look into the varargs bug
<headius> the varargs issue also works in 19
<headius> ahh this was due to an optimization byteit101 did for populating a primitive arg array I think
<byteit101[m]> I didn't optimize primitive arrays, just object arrays, IIRC
<headius> yeah it is just broken
<headius> if (varArrayType.isPrimitive()) {
<headius> varArrayType for a primitive int array is [I
<headius> it should be component type
<headius> I missed it too
<headius> apparently we have no tests for calling primitive varargs ðŸĪŠ
<headius> honestly how does that get this far without being caught
<byteit101[m]> I remember I didn't want to figure out what to test for the primitive stuff, so just stuck that if there. Oops :-)
<headius> must not be many primitive varargs arrays out there
<headius> not sure I ever would have caught that just reading the code
<headius> trivial fix anyway and stepping found it right away
<byteit101[m]> So I correct my earlier statement: I didn't think I optimized primitive arrays
<headius> you didn't
<headius> you shortcutted object arrays since they don't have to use reflection
<headius> but because of this primitive arrays also did it
<headius> optimizing primitive arrays would require calling out to something that could cast to e.g. int[] and set directly (which might be worth it, except apparently this is really uncommon in varargs!)
<headius> zero hits for any primitive... in JI specs
<headius> let's call this a good thing 😀
<headius> varargs fix and tests: https://github.com/jruby/jruby/pull/6834
<enebo[m]> yuck on the string problem
<enebo[m]> JRuby always worked from the premise we make an encoding for a bytelist and add data as we build up the string
<enebo[m]> MRI passes in the encoding for the lex and another enc variable for whatever the string will be
<enebo[m]> It starts out as null
<enebo[m]> So when it enters the unicode escape processing it still has no encoding at all
<enebo[m]> but it sees it is processing unicode and sets it to utf-8
<enebo[m]> in the case it does not see a condition like that it will stay null
<enebo[m]> at the end of the method it will say "what no encoding yet...let's just use the other encoding we passed in"
<enebo[m]> So the problem I slowly backed into was trying to assume a default encoding up front but that means we end up losing this state to realize we have not figured out the strings encoding yet
<headius> so is it not supposed to be utf-8 when first parsed?
<enebo[m]> This worked for many years too but in fixing the last known parsing errors I finally realize I am at a dead end
<headius> I don't think I'm following... if the bytes are valid Windows-31J why does it fail?
<headius> or are we doing something with the bytes that makes them unicode only?
<enebo[m]> It reads a file which has unicode text escapes and quotes
<enebo[m]> that is all ascii
<enebo[m]> it then evals that ascii string which makes a utf-8 string
<enebo[m]> but because we set encoding to windows-31J it assumes the string will starts as a windows-31J string and not a null one until we determine what it should be
<enebo[m]> So I will need to change strterm to mimic this which means pushing through an extra nullable encoding field OR some state saying we have not figured that out yet
<enebo[m]> In both cases they end up as out params
<enebo[m]> which I suspect is why I did it by leaning on bytelist
<basshelal[m]> headius
<basshelal[m]> Sorry if you're busy but see this in case you haven't
<basshelal[m]> The guys at Jenkins use JNR-FFI, that's freaking awesome and freaking terrifying 😂🙃
<headius> basshelal: thanks, had not seen that
<headius> doesn't sound critical but I know maven projects get cranky if you have differing versions
<headius> enebo: ohhhh
<headius> if it's all ascii despite the escapes why is it not being encoded as US-ASCII at that point?
<headius> I'm still not clear why it isn't able to be forced to W31J if the bytes are valid as W31J
<enebo[m]> well it is likely whatever is the default external encoding at that point
<enebo[m]> the reason is when I make strings in a window31J file I just initially set the encoding to that
<headius> ohhhh
<enebo[m]> Which is normally what it is
<headius> so it gets marked as W31J but it has unicode escapes in it
<enebo[m]> but in the case of an explicit unicode escape it must be utf-8 then
<headius> and we don't know how to handle unicode escapes in W31J string
<enebo[m]> well at the point of the error we don't know that we did not have a reason for it to be windows-31j or whether it is fresh
<enebo[m]> If there were already valid windows-31J we would have set it to that already then run into a rnado unicode escape and in that case we should error
<enebo[m]> in this case there was nothing but unicode escapes so it should have not errored and become a utf-8 string
<enebo[m]> So the good news is I completely understand why it is broken
<enebo[m]> I just dislike the fix I will have to make
<headius> oops, I need to revert ostruct tests too
<headius> once again I am burned by committing to master
<enebo[m]> yeah
<enebo[m]> I am getting more into the groove but near release it is much more tempting
<headius> yeah
<enebo[m]> I do totally know the fix but I have to audit all paths to see which ones may decide to set the encoding
<headius> should clean up the test failures
<enebo[m]> I have found two calls in MRI which ignore the encoding param they pass in
<headius> son of a
<headius> I think it's time I filed a bug on ruby-core about all these mismatched versions
<headius> they need to stop committing directly to ruby when these libraries have already been gemified.
<enebo[m]> we shall see how this goes...afk a little bit
<headius> like and subscribe
<enebo[m]> ok the Nothing.java did somehow make it into a jar :(
<headius> Ah figures