<headius> donv: Hey there I am working now
<headius> catching up messages
<headius> kares: yeah optimizing super call paths is at the top of my list... it shows up in every profile and if it's showing up then it also means these are heavily hit methods that aren't inlining either
<headius> I am interested in seeing this megamorphic site
<headius> we may need to move toward some method splitting to avoid megamorphic sites like that, if it's truly mega
<headius> (but splitting can only do so much... we can't split dozens or hundreds of ways)
<headius> donv: Best option now would be to check in what you have and I will look at the guide. It sounds like you have done a lot of great work!
<donv[m]> <headius> "donv: Best option now would be..." <- Thanks! I will check in what I have for JRuby 9.4 on a `jruby_9_4` branch. Master works with JRuby 9.2.
<donv[m]> <headius> "catching up messages" <- I will be working on this today for the next few hours.
<headius> ok I am just putting together slides but I could give this a shot in a few minutes
<headius> if you will be around for a while then I'll refocus on that for a bit
<headius> pulling now
<donv[m]> headius: Not pushed, yet...
<donv[m]> headius: Pushed now.
<donv[m]> <headius> "pulling now" <- org.jruby.util.io.ModeFlags is using com.sun.nio.file.ExtendedOpenOption.NOSHARE_DELETE which is missing. Catching `java.lang.NoSuchFieldError` looks like the solution for this.
<donv[m]> Added the missing files and added the catch. I believe this is not the long term solution. Just trying to get it to work :)
<headius> ok
<headius> yeah that's fine
<headius> I have run into that being problematic elsewhere too
<donv[m]> headius: ```
<headius> it's not a public API
<headius> I think we may only be using that for Windows so it could be masked
<donv[m]> java.lang.NoSuchMethodError: No virtual method setFile(Ljava/lang/String;)V in class Lorg/jruby/parser/StaticScope; or its super classes (declaration of 'org.jruby.parser.StaticScope' appears in /data/app/~~V8EUUBTMIR9QR_uVy44DOA==/org.ruboto.jruby9k_poc-Pp_Gk4iyVaIe8GJi6DJahg==/base.apk!classes8.dex)
<headius> is this pushed work in ruboto or jruby9k_poc
<headius> that seems to be a mismatched version... we would not be able to build if that method were missing
<donv[m]> OK, I'll update it.
<headius> I see your pushes on Ruboto master, is this the right place?
<donv[m]> I am working on JRuby9k_POC to get that to work, then backport to ruboto.
<headius> ok
<donv[m]> Good plan?
<donv[m]> Oh, I have a meeting. AFK for half an hour.
<headius> ok I'll be here
<headius> yes good plan
<headius> goal was always to get the 9k poc to be Ruboto proper anyway
<headius> heh IntelliJ start up Gradle right way when I opened the project
<headius> so that's a good sign for it fitting into Android build world
<headius> ok working through the guide
<headius> can you build it for 12 and have it run on 13? or does that security issue stop it?
<headius> ok I assume the guide you mean is the one in the ruboto repo
<headius> and JRuby9k_POC is just an app that uses Ruboto and updates some stuff, am I remembering the structure right?
<headius> ok I managed to gradle assemble after updating target SDK
<headius> oh of course it won't load on my phone though :-)
<headius> how are you reproducing these issues with 13? I can try to hack on it a bit
<headius> I should say now I know almost nothing about building Android stuff, but I want to learn and make this work for me and others
<donv[m]> headius: I'm back.
<headius> hi there!
<headius> I'm working through this... it has been a long time since I did anything with android
<donv[m]> <headius> "and JRuby9k_POC is just an app..." <- Yes.
<headius> I just bumped target SDK to 33 to see if I can get it to load or error on my phone
<donv[m]> <headius> "how are you reproducing these..." <- I use the emulator in Android Studio.
<headius> right, I figured that's the way to avoid phone security issues but I want to try anyway :-)
<headius> I have the POC up in Android Studio now
<donv[m]> Running?
<headius> sorry no, just up :-) I realized I was still in intellij
<headius> getting emulator running now
<donv[m]> java.lang.invoke.WrongMethodTypeException: Expected (org.jruby.parser.StaticScope, org.jruby.runtime.DynamicScope)org.jruby.runtime.scopes.DynamicScope0 but was (org.jruby.parser.StaticScope, org.jruby.runtime.DynamicScope)org.jruby.runtime.DynamicScope
<donv[m]> at org.jruby.parser.StaticScope.construct(StaticScope.java:192)
<donv[m]> at org.jruby.runtime.DynamicScope.newDynamicScope(DynamicScope.java:54)
<donv[m]> at org.jruby.runtime.DynamicScope.newDynamicScope(DynamicScope.java:97)
<donv[m]> Sorry, forgot to clean the project.
<headius> gradle complains about needing 17, where do I set that in the IDE?
<donv[m]> A fresh version of StaticScope.java got me past the previous error.
<headius> ok good
<headius> oh I found it in gradle settings
<headius> yay I got it to run and crash
<donv[m]> java.lang.invoke.WrongMethodTypeException: Expected (org.jruby.parser.StaticScope, org.jruby.runtime.DynamicScope)org.jruby.runtime.scopes.DynamicScope0 but was (org.jruby.parser.StaticScope, org.jruby.runtime.DynamicScope)org.jruby.runtime.DynamicScope
<donv[m]> at org.jruby.runtime.DynamicScope.newDynamicScope(DynamicScope.java:54)
<donv[m]> at org.jruby.runtime.DynamicScope.newDynamicScope(DynamicScope.java:97)
<donv[m]> at org.jruby.parser.StaticScope.construct(StaticScope.java:192)
<headius> oops I was on master... now on jruby_9_4 and repeating
<donv[m]> Maybe old DynamicScopeGenerator ?
<headius> that should only be used at build time now
<headius> "should"
<headius> I'm getting a duplicate class failure during build
<headius> on the branch
<donv[m]> digest?
<headius> yeah
<donv[m]> clean project.
<donv[m]> Build => Clean Project in the menu
<headius> aha
<headius> so it is picking up old stuff from master build
<donv[m]> Yeah, I wish it didn't.
<headius> we can come up with a better way, probably hook gradle clean into scrubbing temp files from jruby update?
<donv[m]> I think we can, but will take some digging.
<donv[m]> Hey: Found a cleanup line in app/update_jruby_jar.sh Will add removing digest there.
<headius> nice
<headius> that clean did not seem to work but I saw a rogue digest.jar in git status
<headius> yeah progressing now
<headius> wow desugar takes forever
<headius> Caused by: java.lang.NoSuchMethodException: java.lang.invoke.MethodHandles.identity [boolean]
<headius> woot
<donv[m]> That is the api 33 security limitation
<headius> is it method handles in general that are a problem?
<headius> identity just returns a method handle that returns its single argument
<headius> it's probably the most basic handle
<donv[m]> Hey!
<headius> sure seems like an android bug... it's not even the MethodHandles.identity that we call
<headius> it's further into the stack
<headius> this is happening setting up a SwitchPoint... in <clinit>
<headius> not even constructing one, just loading the class!
<donv[m]> I just started the POC with JRuby 9.4.2.0!
<headius> woah what did you change?
<donv[m]> Pushing changes now...
<headius> exciting!
<donv[m]> Basicly deleted the custom DynamicScopeGenerator.java
<donv[m]> pushed.
<donv[m]> ActiveRecord fails, but hey, it starts!
<headius> that's no problem, we can demo stuff without AR and still be able to show it off!
<headius> (but I bet the AR issue will not be too hard)
<headius> does not seem to be pasting right
<headius> oh I see, side scroll
<headius> hahah
<headius> go figure it doesn't like my mangled names :-)
<headius> actually I think this is from byteit101 support for better Java class extending
<headius> we can improve the naming there
<headius> that is supposed to be legal on any JVM but we run into edge cases like this
<headius> donv: what do I need to do to pick up the patch? update jruby script?
<headius> I'm probably going to get one of the new pixel tablets and start actually doing some ruboto dev
<donv[m]> headius: headius: I think you just need to pull the POC and clean the project.
<headius> ok
<headius> and you got it to load on a 12 emulator I assume, not 13 because of the identity thing
<headius> do you have a link to info about that MethodHandles.identity thing?
<donv[m]> Maybe, 1 sec.
<headius> downloading 12L SDK very slowly
<donv[m]> headius: The relevant log line:
<donv[m]> ```
<headius> ok reading
<donv[m]> Accessing hidden method Ljava/lang/invoke/MethodHandles;->identity(Z)Z (max-target-o, reflection, denied)
<headius> pulled down the csv
<headius> I'm not sure they realize this broke other APIs that are public, or they intended those to not be public either
<headius> this table is difficult to read
<headius> I think this CSV has every method in the system in it? If so then it is promising that most of method handle stuff is not blocked
<donv[m]> I see no difference between api 31 and api 33. Do you?
<headius> hard to tell
<headius> are you looking at the csv's?
<donv[m]> yes
<headius> aha
<headius> I think the problem here is that the MethodHandles.makeIdentity impl uses reflection to access MethodHandles.identity
<headius> they probably don't know about this
<headius> I can't read this chart really but I do see some stuff identity-related being blocked
<headius> reproduction of the issue should be easy... just try to load SwitchPoint in any Android app and it should blow up the same
<headius> so if that's supposed to be a public API then it needs to be patched to allow this access
<headius> woohoo I got it to start on 12L
<headius> JRuby startup time is still legendary but I have to say it has been nice and fast on real hardware
<headius> Ruboto IRB only takes a second or so
<donv[m]> Fast enough for me.
<headius> yeah
<headius> [{"A":42}]
<headius> it works
<headius> I'm giggling here at the coffee shop
<headius> thank you for your help getting this going again... this is promising
<headius> short term I might be able to work around the SwitchPoint thing... it's only used when we enable invokedynamic JIT so I can avoid loading it
<headius> but I'd put money on it being an Android bug
<donv[m]> What about the method naming for AR?
<headius> one of us can patch that or maybe byteit101 can work with us on it
<headius> it's just a matter of demangling it a bit
<headius> I'll have a quick look
<headius> we should maybe update this to update JRuby from the snapshot complete jard
<donv[m]> I'll take a look at the AutoLoad example.
<headius> ok
<donv[m]> Ruby Exception is no longer a Java Exception? print_backtrace is missing.
<headius> Is that something from NativeException? We don't wrap Java exceptions anymore
<headius> they are just Java objects
<headius> so printStackTrace or print_stack_trace should be there
<headius> heh
<headius> private static final SwitchPoint DUMMY = new SwitchPoint();
<headius> that's the line that's breaking it
<headius> it's just an cached, pre-invalidated switchpoint so we're not creating new ones every time we null out the cache
<headius> ok I see this is using indy switchpoint to invalidate global variables always... so that can have a non-indy fallback
<headius> I have generally tried to keep most stuff working without indy but that's not going to scale for long
<headius> no bug reports on this SwitchPoint thing
<donv[m]> ok. The print_backtrace thing is a custom Ruboto method.
<headius> I'm filing a bug
<headius> ah is that added to Ruby Exception?
<headius> if so it just needs to be added to java.lang.Throwable too
<donv[m]> Yes.
<donv[m]> To print cause
<headius> makes sense
<donv[m]> pushed a change.
<donv[m]> Java::ConcurrentRubyExtService.new.basicLoad(JRuby.runtime) needs updating.
<headius> updating?
<headius> we'll have to patch that in concurrent-ruby
<donv[m]> It is getGlobalRuntime() now, right?
<headius> oh I see
<headius> it should move to JRuby::Util.loadService or something
<headius> JRuby::Util.load_ext("org.jruby.ext.socket.SocketLibrary")
<headius> OR it can require 'jruby' to get JRuby.runtime back
<headius> I think
<headius> we started trying to hide some of those self-introspection methods behind require 'jruby'
<headius> this is very exciting
<headius> it looks like we don't have many patches to work out but I will continue using patched classes to try to get past this identity issue
<headius> I had forgotten I modified global vars to always use indy invalidation
<donv[m]> I will start a jruby_9_4 branch on ruboto as well, for use with the POC.
<headius> ok
<headius> I need to go get some dinner but we are on the same page now and I can start helping to patch issues
<headius> I'll handle that android issue too
<headius> I'm sure I'll run into it via other parts of Jruby
<donv[m]> I'll be here. Bin appetite!
<donv[m]> Bon!
<headius> and I'm back... full of spicy ramen and sitting in a quiet corner at Mikkeller
<headius> I'm going to try to find the offending code in Android for that identity issue
<headius> hmm hard to find
<headius> donv: do you know how to find the actual JDK sources that are in each android level?
<headius> I had it download sources but these are just the stub sources and don't have any guts
<donv[m]> Very long time since I hunted Android source, sorry.
<headius> ok
<headius> I know at some point they started switching to the OpenJDK sources but I can't find an equivalent source file for this
<headius> aha
<headius> they patched this source
<headius> so they screwed it up
<donv[m]> Got the Autoload examples working.
<headius> nice!
<headius> I'm splunking in android source for a bit
<headius> spelunking
<donv[m]> pushed the changes
<headius> the original change is from 2016
<headius> I'm guessing they just didn't audit these changes when they started locking down access to these APIs. This commit basically expands MethodHandles.identity across all primitive types and then uses a MethodHandles::Lookup object to get a reference to them. They need to allow that operation, or else acquire those references some other way. This is definitely their bug.
<donv[m]> ok
<donv[m]> Are you hopeful it will be fixed?
<donv[m]> If we fix the AR class name issue, I think the POC is complete for API 27-32. I would love to merge that into master.
<headius> I am hopeful yes
<headius> ok I'll pivot and look into that naming
<headius> not sure what part it hates most
<headius> I will see that if I just run the AR demo yes?
<donv[m]> yes
<headius> I don't think the formatting of the name has any importance, like it does for jitted Ruby methods (I have to embed enough info to create a Ruby stack trace)
<headius> hmm I get a different error in my env
<headius> LoadError: no such file to load -- jar:/data/app/~~G7X9n93b4ObagfZxM4IUPg==/org.ruboto.jruby9k_poc-CH2fFUdpkZ9m9WaSQPl-7w==/base.apk!/active_model/validations/absence.rb
<headius> oh I see some gemfile change?
<headius> didn't seem to change anything bundling
<headius> ok now I see it
<headius> I'm not sure what part of this it dislikes
<donv[m]> 1 sec
<donv[m]> You get the same error now?
<headius> yes
<headius> rebundling and rebuilding seemed to get it
<headius> I think it's the comma
<headius> the = and , in that name are valid in JVM spec and are how we encode some special characters
<headius> ah, it's how we encode a . character
<headius> well the good news is we just have to adjust this mangler to use less exotic characters
<headius> debating whether this is worth filing a as a bug, because it's pretty deep in the dex spec
<donv[m]> Can't hurt, can it?
<headius> yeah just not sure how eager I am to fight for it :-)
<donv[m]> :)
<headius> I think we should just replace this mangler with some higher multilingual plane characters unlikely to come up in identifiers
<headius> according to this down they allow U+10000 to U+10FFFF so we have some options
<headius> s/down/doc/
<donv[m]> Sounds good.
<headius> I think I got past that by encoding it with Klingon characters (really just using the ConScript area of Unicode that's set aside for private uses) but something else seems to break when loading the activity now
<headius> not sure if I broke it
<headius> nope, I broke it
<donv[m]> I checked in a couple of small changes. Better layout for the buttons. It should not affect you.
<headius> ok
<headius> donv: getting late here for me so I will give you this patch... it seems like it should be ok, but something I'm doing here now breaks the child activities (json, activerecord, etc) so that they don't even launch properly
<headius> I probably screwed something up here needed to load the new subclass once it has been created
<donv[m]> I'll try it.
<headius> you get the idea though
<headius> some change like this should make it ok for dex
<donv[m]> Can we include a full java class instead of patching JRuby?
<headius> what do you mean
<headius> sorry the actual change I made in here starts around DANGEROUS_CHARS
<donv[m]> The patch patches "JavaNameMangler.java", right? Can we include your version of "JavaNameMangler.java" in Ruboto?
<headius> this patch does that
<headius> in the POC
<headius> I'm not married to the existing mangling either... it's just what John Rose from the Hotspot team came up with
<donv[m]> OK, i'll try it. End of my day here, as well. Have a good night!
<headius> Thank you for your work!
<headius> I will see Michael on Wednesday some time
<donv[m]> Thank you!
<donv[m]> I'd be happy to help you set up for the presentation.
<headius> it won't be much! a few slides to show it working
<headius> Michael may be doing more, but I'll find out his plan soon
<byteit101[m]> I'm on going to be on vacation until later this month, but I can look at mangling then
<headius> byteit101: no problem I think if we can figure out the mangling logic that works for dex it's fine
<headius> it seems to just be a simple name problem
<byteit101[m]> sounds good