<headius> Yeah should be possible
sagax has quit [Read error: Connection reset by peer]
drbobbeaty has quit [Quit: Textual IRC Client: www.textualapp.com]
drbobbeaty has joined #jruby
<headius> g'da
<headius> ack... g'day
<headius> enebo: shyouhei replied that if it were his choice right now he would stick with fileutils 1.4.1 rather than try to pull in a newer version
<headius> after Thursday there may be a 1.4.2 but we are probably safe deferring the upgrade to 9.3.1
<enebo[m]> ok
<headius> (the change is fairly minor anyway)
<headius> I will punt the issue to 9.3.1
<enebo[m]> so I am trying to change Mavenfile in jruby-jars so it runs without the asm error
<headius> I didn't see the error, is it in the email to mkristian?
<enebo[m]> No I discovered it after that
<enebo[m]> I posted a gist showing the error yesterday but it is ClassNotFound on asm OpCodes
<headius> oh yeah the renaming thing
<enebo[m]> I speculated it may have to do with core and base being included in packaging
<enebo[m]> So I have been trying to exclude but that itself I have not figured out yet
<headius> I guess core would be doing the renaming since it does the shading
<headius> if base is getting pulled in as well and replacing those classes it would go back to original name but asm still wrong
<enebo[m]> we must not run jruby-jars in any tests
<headius> that doesn't surprise me
<headius> I have an M1 runner configured and running thanks to MacStadium's OSS program
<headius> going to see if I can toss a job to it so we can start shoring up compat there
<headius> (release notes on hold since we are still working through release issues)
<headius> I don't think this will pass but that was easier than I thought... install Java 11, set up the runner and run it, and throw a job into workflows
<headius> rake test:jruby is actually running, albeit with (expected) errors
<headius> yeah native support is not loading, that aligns with reports
<headius> I started filling out release notes with a few major items: https://github.com/jruby/jruby/releases/tag/untagged-91cd42b1ba6fa7f36259
<headius> anyone that thinks they need to see this (or who might be able to help), let me know if you can't see it and I will try to open up the permissions
<headius> i.e. anyone that contributed a non-trivial amount to this release or just wants to help read through issues and add bullets
<headius> maybe I should move this into an issue 🤔
<headius> yes I decided that would be easier
<enebo[m]> good news in a sense....
<enebo[m]> java -jar ./maven/jruby-complete/target/jruby-complete-9.3.0.0.jar -S gem list
<enebo[m]> Exception in thread "main" java.lang.NoClassDefFoundError: org/objectweb/asm/ClassVisitor
<enebo[m]> Not sure why I didn't notice this earlier but our jruby-complete bit is just broken
<enebo[m]> The other thing I noticed in jruby-complete:
<enebo[m]> # we have no sources and attach the sources and javadocs from jruby-core
<enebo[m]> # later in the build so IDE can use them
<enebo[m]> plugin( :source, 'skipSource' => 'true' )
<enebo[m]> haha
<enebo[m]> So in theory this plugin line was made because jruby-core has no source ... which is the first problem we ran into
<headius> aha ok
<headius> at least this is making some sense
<enebo[m]> if only it actually skipped source though :)
<enebo[m]> I guess it is skipping it's own source but not jruby-core not having source
<enebo[m]> perhaps I can solve it in jruby-core with that line
<enebo[m]> I will give that a try
<enebo[m]> # YUUGE
<enebo[m]> ## YUUGER
<enebo[m]> ok any amount of # is bold/h1
<enebo[m]> hmm. ok no stickers on channel
<enebo[m]> funny I wonder what that looks like from irc bridge
<enebo[m]> yay fixed first problem...now to fix what is with jruby-complete
<enebo[m]> LOL. mvn -Pcomplete makes a jruby-complete.jar which contains NO classes from jruby but if performed as part of the release build must
<enebo[m]> Trying again with a clean build
<enebo[m]> I think it might be time to migrate to a new build system. If for no other reason that I understand it
<enebo[m]> I wish buildr had taken off
<enebo[m]> ok looks like complete requires core has been built first
<enebo[m]> it gets its contents from the core target .jar
<enebo[m]> which makes me wonder if core is actually usable now. jruby-complete it just stuffing the same contents into it
<enebo[m]> ok hahaha mvn -Pcore will make an empty jar with only the poms in it.
<enebo[m]> So I guess mvn -Pbase -Pcore -Pcomplete ?
<enebo[m]> Hmm -Pbase will do core and stdlib but if I look at target jar made for core it only contains the pom files
<enebo[m]> lib/jruby.jar contains no shaded stuff that I see but the contents of jruby-base and lib/jruby.jar differ too. I am rerunning this but I am pretty confused and just spamming the channel with running commentary
<enebo[m]> related up a couple of layers in the onion: https://app.travis-ci.com/github/jruby/jruby/jobs/537386439
<enebo[m]> We stopped passing this job a while ago because there is no org/jruby/Main in jruby-complete.jar anymore. It is running Pjruby_complete_jar_extended vs -Pcomplete but more or less same issue
<enebo[m]> yow...that job has not build jruby-complete for at least a year. It has been a rough 18 months
<byteit101[m]> headius: in the release notes it may be good to mention my first PR related to annotations for JI
<byteit101[m]> #4165
<byteit101[m]> (or 6141 if you want the PR vs the issue number)
<enebo[m]> ok jruby-core makes an empty file but lib/jruby.jar is supposedly the result of shading base...but it appears to have no shading. jruby-complete I think is loading the empty jruby-core in shaded and not lib/jruby.jar
<enebo[m]> ok yeah jruby-core with no source is being copied to .m2/repository
<headius> byteit101: ah yeah good call
<headius> enebo: ouch
<enebo[m]> yeah so I typed a lot so I am not sure if you read back but I can catch you up if not
<enebo[m]> -Pcomplete (and more extended phase for travis test) has not worked in over a year (logs get lost at that point)
<enebo[m]> but it does not work because it is using jruby-core and the installed target has no source in it
<enebo[m]> weirdly when we do a release build we will get a jruby-complete which has classes in it but none of them are shaded
<enebo[m]> when core builds it does make lib/jruby.jar which has what we need but none of those are shaded
<enebo[m]> headius: Should lib/jruby.jar be shaded or not?
<enebo[m]> I assumed it would still be jruby-core version of jar but I don't remember if we switched to jruby-base
<enebo[m]> s/has no source/has no jruby classes/
<headius> lib/jruby.jar should be shaded for now
<headius> when we drop Java 8 it won't be and we will use module path logic
<headius> I will have to catch up a bit later, family duties for a while
<enebo[m]> ok. I am making some progress in seeing threads to pull on :)
<enebo[m]> and since I just wrote it I see that asm is actually shaded so ignore that claim that we were not shading
<headius> Ok good
<headius> FWIW none of this surprises me much... I knew the new artifact would probably need some tweaking. Well get it figured out this week
<enebo[m]> I feel like if we can just make sure that jruby-core is written to proper target location things may just work
<enebo[m]> we definitely need it to be there so we don't push an empty artifact up
<enebo[m]> I am very bold. I think I fixed the build and just pushed it to master
<enebo[m]> I removed shading directly to lib/jruby.jar so it writes to proper target location then I use another plugin to copy that to lib/jruby.jar
<headius> That does seem better
<enebo[m]> hmm
<enebo[m]> so jruby-core is set to being <10M? it is current 15M so this job has been looking at the empty jar I guess
<enebo[m]> jruby-complete is 2M larger than before but has not actually built in over a year
<enebo[m]> I think I will look at sizes of 9.2
<headius> It has grown but did not expect 5mb
<enebo[m]> yeah so that is what I am mainly concerned about
<enebo[m]> 26808 -rw-------. 1 enebo enebo 27447709 Jun 15 09:02 ../../releases/9.2.19.0/jruby-complete-9.2.19.0.jar
<enebo[m]> Weirdly this is actually larger than our check on master
<headius> Wow
<headius> Oh complete
<headius> Bug but yeah
<headius> Big
<headius> We have had problems with too much getting in these jars sometimes
<headius> Usually it is obvious if you unpack and look through it
<enebo[m]> yay...that failing job using jruby-complete is now running tests
<enebo[m]> It has not run for over a year so hopefully it runs to completion
<enebo[m]> jruby-core-9.2.19.0.jar 2021-06-15 13:48 10319585
<enebo[m]> ok so perhaps there is something unwanted in that
<enebo[m]> but it was there before this last commit as well
<headius> Yeah that is quite a big difference
<enebo[m]> So 2 other errors one is it cannot find JavaFields as a fixture for a test but that is in the job which has not in a long time
<enebo[m]> The other error is that stupid ScriptingContainerTest on J9
<enebo[m]> system ~/work/jruby master 1195% ls -l lib/jruby.jar
<enebo[m]> system ~/work/jruby master 1196% -rw-r--r-- enebo/enebo 16196214 jruby-9.2.19.0/lib/jruby.jar
<enebo[m]> 15284 -rw-r--r--. 1 enebo enebo 15650418 Sep 14 15:45 lib/jruby.jar
<enebo[m]> headius: ok now I am confused. jruby-bin for 9.2.19.0 is virtually the same size as 9.3.0.0 but jruby-core on sonatype is quite a bit smaller
<enebo[m]> ok this is amusing or not depending on your sense of humor
<enebo[m]> 9.2.19.0 and 9.3 have the same problem
<enebo[m]> Although my changes made it more obvious for 9.3
<enebo[m]> in addition to having the shaded versions of classes in our bin/dist we also still have all the original classes in the uber jar
<enebo[m]> however 9.3 maven works now that will end up being contained in jruby-core now
<enebo[m]> lopex: is tables/*.bin used at runtime to extra jcodings data?
<lopex[m]> enebo: yes
<enebo[m]> should it end up jruby-core?
<lopex[m]> in a jar ? yes
<enebo[m]> ok. I don't think it is in any of our jruby-core releases
<lopex[m]> hmm
<enebo[m]> I am questioning whether jruby-core even works atm
<enebo[m]> It is also missing other non-relocated packages
<lopex[m]> it is in lataest jruby.jar release
<enebo[m]> yeah but you mean within jruby-bin.tar.gz
<enebo[m]> the lib/jruby.jar does contain them within the bin dist
<enebo[m]> but the jruby-core artifact does not have them
<enebo[m]> (and not just tables...it is missing other stuff too)
<lopex[m]> ok
<enebo[m]> I was just fixing a build issue and realized once I "fixed" things jruby-core was ~5MB larger
<enebo[m]> java -cp jruby-core-9.2.19.0.jar org.jruby.Main -e 1
<enebo[m]> Exception in thread "main" java.lang.NoClassDefFoundError: com/headius/options/Option
<headius> Can you push what you have to a branch and I might look at it more tonight?
<enebo[m]> It is all pushed
<headius> Ok
<enebo[m]> but I believe the issue is that jruby-core has never shipped shaded libs in it to maven
<enebo[m]> Or if it has it has not been for a while
<headius> So it may not have been quite right to begin with
<enebo[m]> master does now but that is why it is so much bigger
<enebo[m]> things like tables from jcodings makes some heft
<enebo[m]> So if you look there were 3 problems:
<enebo[m]> 1. checkVersion probably is using too low of values for sanity checks
<enebo[m]> 2. on J9 scriptingcontainertest.testGetEnvironment threw a NPE trying to call instance.terminate()????
<headius> Weird
<enebo[m]> 3. Java::JavaLang::ClassNotFoundException: java_integration.fixtures.JavaFields
<enebo[m]> This last one is weird because the .java file is there but this was running complete tests which have not run in over a year
<enebo[m]> I find both 2 and 3 pretty weird but for all I know we have something flakey J9 is hitting like a shared container
<enebo[m]> tbh I was a little down about debugging this but it ended up being a bit more tractable than I thought
<enebo[m]> Ok #2 passed on next run so that is pretty weird
<enebo[m]> #3 seems to be that we are not compiling the java_integration fixtures any more as part of this run
<enebo[m]> going to hang it for the night
<headius> ok I'll see what I can do
<headius> confirmed that jruby-core is being compressed at least, so that isn't it
<headius> some of this could be the additional platforms in jffi, we have added several
<headius> but I thought 9.2 was mostly up to date on the same jffi
<headius> going to diff against 9.2's core jar
<headius> aha I think I have it
<headius> the check-versions script is expecting to check an unshaded jar file
<headius> the actual shaded jar has gotten smaller
<headius> the complete jar is a bit bigger but that may be due to more stdlib being in gems... I will investigate though
<headius> [ERROR] /Users/headius/projects/jruby-9.2/core/src/main/java/org/jruby/RubyEnumerable.java:171: error: invalid use of a restricted identifier 'yield'
<headius> [ERROR] return yield(context, packEnumValues(context, args));
<headius> [ERROR] ^
<headius> gonna have to look into that before we can build and deploy on 17
<headius> the complete jar appears to be including anything installed in the working copy, so for me it pulls in things like bin/benchmark_driver, bin/httpclient and so on
<headius> that needs to be checked against a clean working copy
<headius> only one new gem, for the new io-console jruby support, but that shouldn't be too much bigger
<headius> oh, and IRB... but I don't see the gem file being included in the complete jar
<headius> some of these are default gems but there are several new ones in 9.2 which would increase the size of complete jar
<headius> bundler is now shipped
<headius> csv is newer with more files
<headius> fileutils is a gem now
<headius> the 9.2 complete jar does not appear to shade the x86_64 and aarch64 assembler libraries from JNR
<headius> that's about it
<headius> only stuff that didn't seem to belong was from bin/ so the script is a little too eager there
<headius> that's probably because we kept missing files and modified it to include everything
<headius> here'
<headius> here's the diffs for core and complete from 9.2 to 9.3: https://gist.github.com/headius/2e43ef44ab3ed49d53b511e5587650b6
<headius> nothing seems particularly odd in 9.3 complete jar so I think bumping it to 30MB is probably appropriate
<headius> that makes sense because the jruby-jars gem size is also 30MB and should include basically the same things
<enebo[m]> ok cool. yeah we always build on a shallow clone so how we include should be fine
<headius> yeah seems like the sizes are no worry... I am pushing a bump for the check_versions script
<headius> the complete jar may perform better in JNR because it actually includes the assembler for jni stubs now
<enebo[m]> headius: seriously makes me wonder if anyone uses jruby-core as an artifact too
<headius> the fact that the core jar shrunk is AMAZING to me though
<headius> I don't know if that has ever happened
<headius> we yanked out nailgun and the parser and ripper parser both lost a ton of inner classes
<headius> and many other inner classes replaced by lambdas
<enebo[m]> yeah
<enebo[m]> lets make the entire codebase into lambdas and we will only load one class
<headius> sweet
<headius> one big file with lambdas
<headius> so #2 could just be openj9 glitchiness.. I'm not going to dwell on that right now
<headius> if it passed after that it isn't a priority
<enebo[m]> yeah it happened once and then didn't
<enebo[m]> tests for complete lost whatever fixture compilation
<headius> JI tests passed in the last run, where did you see a problem?
<headius> the only thing that failed unexpectedly in the latest travis run was check_versions
<enebo[m]> I rm'd my compiled fixtured and I see the same error
<enebo[m]> in the expected failure job running -Pcomplete_*
<headius> oh but that has been failing for some time... if that is the only problem now it may be fixable
<enebo[m]> once it makes it past that section there may be other errors but so long as we have them running we may as well make at least that section pass
<headius> it was unclear why it was failing before but I didn't think it was something that easy
<enebo[m]> It seems to be the only CI test we have to see if complete is working
<enebo[m]> I always check before release manually but catching it is nice
<headius> it is some maven thingy that mkristian created to test complete and it stopped working at some point
<headius> I would bet it just doesn't compile those sources because they are compiled in the rake target
<enebo[m]> it was because jruby-core as an artifact was a 4k file with nothing in it
<headius> that could be fixed by having them compiled by maven or something
<enebo[m]> so jruby-complete put that in and have no classes in it
<headius> right ok
<headius> so that is not new but if we want to get it working I can spend some time on it
<enebo[m]> So all we need to fix there is getting the test java fixtures compiling
<headius> I am pretty sure those files are only getting compiled by the rake target because they're in a weird place
<headius> spec/ji/fixtures is not part of any maven module's test sources
<enebo[m]> maybe we can just comment those out and open an issue too
<enebo[m]> I would like to see once we make it past those first section whether the other crud in there is missing stuff. Once we pass the maven tests I think it does some app server things
<enebo[m]> Maven really makes some class of activities painful
<headius> yeah we could skip the JI tests for the extended complete thing for now