subbu has joined #jruby
subbu has quit [Quit: Leaving]
subbu has joined #jruby
subbu has quit [Quit: Leaving]
drbobbeaty has quit [Read error: Connection reset by peer]
<headius> Good morning!
<nilsding> good... afternoon! :)
<headius> hooray!
<headius> let
<headius> let's fix some bugs
<headius> and learn how to type
drbobbeaty has joined #jruby
<headius> at_exit fix is merged... I also modified the rubyspec run to use MRI-style backtraces because these specs check process output
<headius> hmm I just realized we could eliminate some frame pushes by detecting a tail call and just reusing the current frame
<headius> dunno if that would improve perf very much, since we still need to populate it
subbu has joined #jruby
<headius> enebo: hey you did some estimates of methods in a simple rails app at some point... did we ever do any measurements of how many JVM classes we generate?
<headius> I got an email from someone working on reducing the object header sizes in OpenJDK and that compression can vary depending on how many maximum classes the JVM can handle... lower would make the header smaller, potentially (so presumably this is down to a bit width for the class pointer)
fzakaria[m] has quit [Ping timeout: 260 seconds]
jswenson[m] has quit [Ping timeout: 260 seconds]
headius has quit [Ping timeout: 260 seconds]
BlaneDabneyGitte has quit [Ping timeout: 260 seconds]
Bi[m] has quit [Ping timeout: 260 seconds]
kares[m] has quit [Ping timeout: 260 seconds]
fzakaria[m] has joined #jruby
headius has joined #jruby
<enebo[m]1> headius: I have no record of what I found out but I think for class count I just ran jvisualvm and connected after a while
<enebo[m]1> I do recall that in a simple rails app 84% of the loaded methods were not called
<enebo[m]1> which means 16% would eventually JIT
<enebo[m]1> but that was an older Rails and pretty informal since it was I think scaffolded app (I might have did it with redmine as well)
<headius> the other backtrace bug guy confirmed it's fixed with snapshot
<enebo[m]1> I have been thinking about doing some analysis on literal blocks
<headius> yeah so I gave this guy a gross estimate of 5000 methods getting jitted with 100 indy call chains in each, so like 505k classes generated of various sizes
<enebo[m]1> I think we have 2 issues now that are needed yet
<headius> I would hope that is a massively high estimate
<enebo[m]1> I did kill a bunch off with the parser change
<headius> and the fallback option here is that a user might have to turn off compressed class pointers or something
<enebo[m]1> that was like 1000 less classes :)
<headius> you killed off the .class files but they probably still generate classes for lambda internally
jswenson[m] has joined #jruby
<headius> clearly indy/lambda are going to be the real class-suckers
<enebo[m]1> oh well in that case we use lambda quite a bit in our code too
<enebo[m]1> not enough to not use them of course
<headius> yeah I didn't consider how many classes we have or lambdas we use but 100*5000 is already the lion's share
<headius> so like, 1M?
<headius> seems super high and the fallback is not terrible if you have a gigantic app
BlaneDabneyGitte has joined #jruby
<enebo[m]1> well I don't know that sounds very high to me but I think most of our code would be from methods right?
<headius> every indy call site will have a chain of LambdaForm attached to it, which eventually get turned into anonymous classes
<headius> potentially one tiny anon class for every indy
<headius> and we use indy for basically everything
<enebo[m]1> so one class per site?
<enebo[m]1> I thought they shared lambda form classes
<headius> anon classes might have different requirements, like maybe they don't have the same object header or something
kares[m] has joined #jruby
<headius> they share the lambda form subtrees and might share classes, yes
Bi[m] has joined #jruby
<enebo[m]1> that would be a good question to ask
<headius> I'm not sure about that part and it keeps changing
<headius> yeah I mentioned that caching might change this
<headius> worst case is that every indy site (method calls, const lookups, literal caches, more) would be a potentially-tiny class
<enebo[m]1> It seems like you cannot answer something when it is per-site or not without some clarification
<headius> many methods will be small and have only a couple and others might have... I don't know, hundreds?
<headius> some of this is limited by how much bytecode we can even load in a single method, and then again by how much hotspot will JIT
<headius> no JIT, no indy specialization and possibly fewer generated classes
<enebo[m]1> I am making an empty rails app just to see what shows up
<headius> yeah cool, thanks... I am triaging other 9.3 issues
<enebo[m]1> empty rails app in console 12,570 in Java 8
<enebo[m]1> headius: did you fix the gem push issue?
<enebo[m]1> headius: Other than that I believe the other important one is us returning wrnog errno on I think symlink? I triaged it and it is hairy
<headius> hah my bug about Hotspot on macOS not having a logical java.library path has been updated by three different people to be assigned to three different components
<headius> otherwise no progress
<headius> I cannot reproduce the gem push issue on either 9.3 or 9.3.1 so I have asked for more info
<enebo[m]1> I plan on adding a bit for line vs column and will fix this today
<headius> I pushed some weakling-pre# gems and they were fine
<headius> basshelal: ^ about that JDK library path issue... at least people are looking at it
<headius> ok
<headius> we should still plan to do 9.3.1 next week even with these other issues in progress
<headius> and a 9.3.2 before RubyConf probably
<enebo[m]1> yeah
<enebo[m]1> I cannot find the other issue and I thought I targetted it already bah
<headius> hopefully I can fix the M1 issues for 9.3.2
<headius> and kares or we can fix up these ossl issues too
<enebo[m]1> 7760 classes with compile.invokedynamic enabled
<enebo[m]1> but perhaps those classes are not being shown in visualvm
<enebo[m]1> lunch
<headius> the openssl cert issue is not new either btw
<headius> at least as far back as 9.2.16
subbu has quit [Quit: Leaving]
<headius> enebo: I do not see much more in issues that we could resolve before Monday. SSL thing predates 9.3, gem push does not reproduce for me (you should try on Linux), and JNR issues continue to need M1 work
<enebo[m]1> ok
<headius> oh I will look over the other things we punted to 9.3.1 because I think that was optimistic for a 2-week flip
<headius> if we had another month maybe
<headius> some of these should be punted to 9.4 given our timeline goals there
<headius> new IRB just worked properly for me on 9.3 so that is promising... I have punted the IRB STDIN.gets bug (kicked around from 2013) to 9.4 since we will switch to new IRB + reline in that release
subbu has joined #jruby
<headius> I am punting bugs like mad
<headius> lots of these were nice to haves for 9.3 but don't apply anymore because they would require larger changes
<headius> if we are pushing 9.4 as a feature release and not a major overhaul then these can slide
<headius> JRuby X can rule the world some time in 2022
<headius> enebo: do you have Windows handy to test
<headius> no updates on this issue for 1.5 years
<headius> I thought we'd have heard more noise if IRB was that broken on Windows
<headius> six issues marked for 9.3.1 are all about IRB and Readline... I punted one because it appears to be working with new IRB and Reline on the 3.0 branch
<byteit101[m]> I haven't had time to reduce the issue on some super JI calls failing yet, but i'm hoping to do so and file an issue this week
<headius> byteit101: yeah cool, whatever reduction you can provide I will jump in and help
<headius> we don't have to feel pressure to fix stuff in 9.3.1 if we can keep up a good pace of updates
<byteit101[m]> Cool, I know you will be much more efficient at solving it than I, as I saw with the last super issue :-)
<headius> hah no worries
fidothe has quit [Ping timeout: 260 seconds]
fidothe has joined #jruby
<ahorek[m]> hey @headius I can confirm that the new reline in 9.4 fixes the problem with arrows
<ahorek[m]> but there's a new error
<ahorek[m]> on_error1 at lib/ruby/stdlib/ripper/lexer.rb:195
<ahorek[m]> parse at org/jruby/ext/ripper/
<ahorek[m]> NoMethodError: undefined method `token' for #<Ripper::Lexer:0x31f5b923>
<headius> aha this is probably ripper stdlib updates I did not copy right, or else missing ripper features at the parser level
<headius> hmm my stdlib copier might not be working right
<headius> oh, figures
<headius> and of course they use it in IRB
<headius> ahorek: shouldn't be hard to add but seriously, what's a standard
<headius> my copy works fine, we just have some minor diffs in this code
<ahorek[m]> does rubygems work on 9.4?
<headius> yes it does, if you are switching from 9.3 you will want to git clean -dxf lib/ruby/stdlib and clean package
<headius> I'm getting you are seeing frozen array errors?
<ahorek[m]> nope, it's just missing on a clean build
<headius> oh hmm... it should install during the build
<headius> ok weird, it is not there
<headius> let me try merging 9.3 and see if that fixes it... the logic to install RG might have come after the 9.4 branch
<headius> confirmed still not installing the files, hmm
<headius> yeah this is strange
<headius> every other gem is fine
<headius> gah, the gem seems to be empty?
<headius> ahorek: hmm I have no explanation... will keep poking at it
<headius> this is weird, something seems wrong with the 3.2.22 rubygems-update
<headius> oooops
<ahorek[m]> require paths
<ahorek[m]> 9.3 /jruby/lib/target/rubygems/gems/rubygems-update-3.2.14*/lib/*
<ahorek[m]> 9.4 /jrubyish/lib/target/rubygems/gems/rubygems-update-3.2.22*/hide_lib_for_update/*
subbu has quit [Quit: Leaving]
<headius> somethign odd is happening
<headius> oh wow
<headius> ok yeah
<headius> this is weird... I install rubygems-update 3.2.22 and it doesn't seem to install right even on 9.3
<headius> woah somehow I got a bad gem_home
<headius> GEM_HOME
<headius> so are they not including lib in require paths now?
<ahorek[m]> aha
<headius> that would be the simple explanation... we only look at the spec for those paths to copy
<headius> we'd have to special case rubygems-update
<headius> yup that's it 🤦‍♂️
<headius> I will fix this
<headius> have to add another option to the gem copier to override require paths
<headius> whew... at least it is not some weird bug of ours... they are literally hiding lib
<ahorek[m]> we do load the actual spec now, it was hardcoded to "lib" before your fix
<headius> so I broke it by unhardcoding it
<headius> yay
<headius> I did see that note.txt and thought I did something odd to produce it
<headius> I have a fix, testing it now
<headius> ok, it installs now but there's that frozen error I mentioned on boot
<headius> etc
<headius> I will push this change and look into that
<headius> pushed
<ahorek[m]> nice thanks
<headius> yeah this is wierd
<headius> I trace the code back and it does seem to be modifying a frozen array
<headius> oh heck
<headius> I broke this I bet
<headius> I probably mucked up Array#dup while removing tainting
<headius> we need to get this branch on CI
<headius> fixing
<headius> that's what I get for working late
<headius> I replaced "dup.flags |= flags & TAINTED_F" with "dup.flags = flase"
<headius> flags
<headius> curse you bitwise math!
<headius> found another one
<headius> so basically I broke two things that hid each other
<headius> the taint breakage prevents RG from loading, but the lib/pom.rb prevents RG from even trying to load
<headius> ahorek: pushed all fixes
<headius> sorry about that 🙈
<ahorek[m]> it's ok :) thanks, but maybe there's something more? this is weird - java.lang.AssertionError: CR_7BIT = 16 but should be 8
<headius> ah, could be more flag problems
<headius> when do you see that
<headius> wow ok
<headius> yup this is due to taint flag going away and the check not being updated
<headius> I didn't realize enebo had these running in GHA... I will mop up my mess
<headius> down side of my oh-so-clever flag allocator perhaps
<headius> ahorek: pushed fix, gem install runs with -J-ea now
<headius> tests seem to be running correctly now
<headius> "square root of 9.0 is 3.0"
<headius> enebo: this warning is weird:
<headius> lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85: warning: parentheses after method name is interpreted as an argument list, not a decomposed argument
<headius> 85 is just `return gem_original_require(path)`