<headius>
hsbt proposed releasing the new timeout on Monday or Tuesday, so we ought to have a greatly improved gem to ship very soon
<kares[m]>
the numbers do look impressive but I wonder what's the MRI baseline, are we on par with the Rails sqlite adapter?
<kares[m]>
it's been a while since I looked into this but I mostly did benchmarks with DBs (on localhost but still) - recall mysql being on par with C-adapters (mysql2)
<kares[m]>
there wasn't that much difference with indy vs non-indy as far as I recall ...
<kares[m]>
guessing smt in JRuby improved or there's a Rails feature in 7.0 that is making the interpreted version super slow 😉
<headius>
kares: non-indy seems to be slightly slower than CRuby 3.1 sqlite3 on this bench, while indy is much faster
<headius>
kares: I have done a small modification to these benchmarks to use IPS and it seems to be working well
<headius>
running with 5 rows per iteration on the update bench gives some nice results
<kares[m]>
great, so (looking at twitter), this is only on Java 17 on older e.g. 11 the numbers aren't that good?
<enebo[m]>
I am wondering if removal of frobnicate on kwargs when rails is using quite a few kwargs is also helping somehow
<enebo[m]>
I am going to compare dispatch perf between 9.3 and master right now
<headius>
it's as good a theory as any
<headius>
kares: I have only been running on 17
<headius>
debating whether to bring both work laptops since the lenovo is more like typical hardware these days, and I may want to run more numbers tonight
<enebo[m]>
headius: It says nothing about indy vs no though
<enebo[m]>
Not concretely anyways
<headius>
yjit in 3.1.2 appears to have practically no effect on the update bench
<headius>
if anything it is slightly slower, but within a margin
<headius>
it has so little an effect I'm not going to bother with numbers on other benchmarks
<headius>
I will have these numbers to show for anyone curious about yjit and rails
<headius>
if you guys could try to reproduce some of these numbers it would make me feel better... we are just so much faster it makes me nervous
<enebo[m]>
In a little bench of calling a:1 with an a: method (no indy) we go from 6.4i/s (this is a while loop calling the method a million times) to 7i/s
<headius>
I will push a branch with the ips change
<kares[m]>
happy to try running some numbers - I am on the avg 64-bit intel with linux
<enebo[m]>
HAHA nice
<enebo[m]>
empty hash ** to method accepting **rest goes from 12.8 i/s to 35 i/s
<headius>
nice
<enebo[m]>
This is actually pretty common in Rails
<headius>
we knew our kwargs were slow before and the new logic helps separate them out, so it's a good theory
<enebo[m]>
This most likely is I currently have IR on callside splitting an if/else and checking for empty kwargs and not passing them at all
<headius>
there's also been a handful of little optimizations the past couple years
<enebo[m]>
I think a big change is we frob last arg for every instr in 9.3 and this does not happen ever now
<enebo[m]>
That mostly is just "oh it is a kwarg' but it was a lot of logic to optimize away
<enebo[m]>
So empty kwargs with indy goes from 14 i/s to 79.6 i/s
<headius>
there's my bench numbers and a couple example command lines for update
<headius>
the indy numbers are off the hook
<headius>
I'm a little disappointed with the yjit numbers... I thought there'd be at least a little gain
<headius>
maybe it doesn't jit
<enebo[m]>
oh lol...I am using Java 8
<headius>
I will try 8 just to see
<enebo[m]>
yjit claims some speed up for running Rails but it may affect some parts of that code differently than others
<enebo[m]>
It was less than 10%
<enebo[m]>
on java 8 the single kwarg to single kwarg is slower with indy on 9.3 and 9.4
<enebo[m]>
Trying Java 17
<headius>
trying 8
<enebo[m]>
hmm a little slower on 17 too but this is a strange call foo(a:1); def foo(a:)
<headius>
8 does seem to be a bit slower
<headius>
maybe 10-15%
<headius>
on update 5x
<headius>
going to do one update run with 17, indy, and parallel GC
<headius>
it has surprisingly little impact on this bench
<enebo[m]>
lol...a bumblebee was in the basment
<enebo[m]>
Got him back outside
<headius>
yeah I would definitely like confirmation that cruby is this slow
<enebo[m]>
So on 17 the **{} -> **rest is 37.7 i/s non-indy and 99 i/s indy
<enebo[m]>
which is way faster than 8 for both
<headius>
it feels like something is wrong
<enebo[m]>
So empty kwargs with indy goes from 14 i/s to 79.6 i/s
<enebo[m]>
^ that was 8 numbers
<headius>
nice
<enebo[m]>
so 17 gives us more than 8
<enebo[m]>
I think a huge problem with this benchmark is it is stuff which will all opt away
<headius>
is it though?
<headius>
it is round tripping through the database driver
<enebo[m]>
It turns into an if statement examining an empty hash and not calling the version which passes it
<enebo[m]>
So virtually all the keyword logic disappears
<enebo[m]>
but this is actually pretty common in Rails where they pass opts down and there tends to not be opts being passed
<headius>
could be CRuby is doing some hard commit after each update?
<enebo[m]>
I mean my microbench on calling methods...I think the insight here is that you are calling through a lot of methods which are def something(**opts) and opts is nothing
<enebo[m]>
In Ruby 3 world and how it is implemented that is equivalent ot calling if (opts.empty) something()
<enebo[m]>
which is a lot cheaper than in 9.3
<headius>
enebo: have you done any work on mysql?
<enebo[m]>
in AR you can see if calls through like 8 layers of calls like that
<headius>
I am a little worried about these sqlite numbers
<enebo[m]>
nope
<enebo[m]>
I was trying to eliminate the last kwargs error which appears to be us losing some kwarg into aliasing send
<enebo[m]>
Although most of my efforts on kwargs has just been making sure the IR design is robust
<enebo[m]>
I am not aware of any ruby->ruby kwargs issue other than with attrassign but that is just missing logic
<enebo[m]>
I did sqlite3 last week because I knew it would be a small amount of work to make it function
<enebo[m]>
I guess I can look at mysql a bit this mroning. It might not be bad. Worst part is getting a working dev env
<headius>
use docker
<enebo[m]>
well I have been historically because I hate havnig dbs running all the time and I forget to disable them otherwise
<headius>
I will present these numbers but with a big caveat that I don't believe we should be this much faster and there may be something keeping CRuby numbers low
<enebo[m]>
The command-line I thnk I have written down
<headius>
unless I can get mysql numbers before tomorrow afternoon
<headius>
I will bring both laptops so I can reproduce numbers if necessary
<enebo[m]>
yeah well if I get it working you will know about it
<enebo[m]>
The funny bit is 7.0 only works with 9.4 so putting out 70.0 means a real release which works against no release
<headius>
heh yeah
<headius>
heh this machine is much faster than my old one... jruby startup time doesn't seem that bad, until I run CRuby
<enebo[m]>
headius: This update being super fast was also in our mpls railsconf talk
<headius>
this update?
<enebo[m]>
It may have gotten even faster now but there was something super weird .. we even had a slide on that
<enebo[m]>
Or perhaps I just talked over the result.
<enebo[m]>
In my mind I thought it was insert and not update but something is happening in our adapter which is doing a lot less work (or delaying commit somehow)
<enebo[m]>
We have not gotten any bugs on the behavior if we are doing it wrong
<headius>
I keep throwing workers at CRuby but can't crack 90% utilization
<headius>
on a testapp post scaffold production mode
<headius>
yeah it is in the neighborhood of 30x on those update numbers
<headius>
we can't be that fast can we?
<enebo[m]>
oh yeah that sounds odd
<enebo[m]>
I was like 5x faster or something like that though
<headius>
hmm
<enebo[m]>
I just remember it being really good and not understanding why to the point I felt like i had to say something
<enebo[m]>
I do remember the actual update code in Ruby is not typical too
<enebo[m]>
hash update of one field in a literal hash
<enebo[m]>
It happens but it is not hwo someone normally updates a record
<headius>
whew... took a second 1m siege run but we are faster than CRuby's best on view post
<headius>
blast that long warmup tail
<headius>
blast it!
<enebo[m]>
My mind keeps thinking of rubygems.org site but I doubt they run on 7 yet
<enebo[m]>
It was a pain to set up a bit since you need to use a couple of fake services like s3 faker
<headius>
3200 req/s vs 2900 for CRuby on this simple case
<enebo[m]>
Also I think we need to run on postgresql
<headius>
still getting hotter but settling in
<headius>
yeah
<enebo[m]>
Our main selling point last time was some gains like you are seeing but we got a lot more reqs processed on the same hardware
<enebo[m]>
We also fell over on a much higher load
<headius>
these are more realistic results... we are faster but not by as much as we could be
<enebo[m]>
Memory also was an overall win once you got to a whole machine being used up
<enebo[m]>
That to me was our biggest point based on cloud
<headius>
yeah I had to spin up 10 workers with 2 threads each to approach 90% utilization on this laptop
<headius>
so that's a GB of memory and we can easily do better than that
<enebo[m]>
with MRI?
<headius>
yeah
<headius>
about 100MB each but hard to know how much of that is shared memory
<enebo[m]>
We were using 1.5G on rubygems.org and the heap was not large at all 200ish MB
<headius>
doesn't matter much, this benchmark should run fine in like 250MB
<enebo[m]>
So I think 17 will improve that overall RSS a lot more
<enebo[m]>
That RSS was the main reason why we did not do even better
<enebo[m]>
Then people came up to tell me that they are custom compile MRI with reduced arena size and even use a different malloc
<enebo[m]>
So having an improved RSS over 2019 will help convince the crowd which is doing that amount to save resources
<headius>
without flags this is running at 2.2GB but there are heap heuristics at play and my machine has 64GB
<headius>
so JDK is just taking a big bite
<enebo[m]>
yeah you need to size heap at a minimum
<headius>
trying some runs with 250
<headius>
the base post scaffold doesn't need much
<enebo[m]>
but I do not think we should have to do anything to see the RSS stuff from oneshotclassloader
<headius>
yeah this is going to be greedy without any flags though, and siege is throwing 25 concurrent at it by default so it will have a lot of live data
<headius>
I should also do some runs that throw like 100 concurrent at it and see if MRI starts to fold
<headius>
this is making me want to get back to perf
<headius>
now on linux I will have access to lower-level tools to see how CPU is handling our code
<headius>
250MB heap tops out around 1GB total memory so there's a lot of off-heap data
<headius>
I think G1 likes a larger heap too
<headius>
ok this is a nice result
<headius>
250 max heap, parallel GC, performs like unlimited heap with G1
<headius>
250 + G1 takes a 30% hit over unlimited + G1
<headius>
these will be interesting numbers to show that there's configurability for different scenarios
<headius>
now I am curious how small I can go
<headius>
that off heap data is frustratingly big though
<enebo[m]>
this was 1.5G a couple of years ago at least for a long running topped out rubygems.org process only hitting one controller
<enebo[m]>
So if it is topping at 1G that is a big improvement (I doubt the other code loaded from rubygems.org is much native memory in the sense we are not hitting it at all)
<headius>
ok 100 is too small
<headius>
yeah clearly we should spend more time looking at off-heap data because that is by far the lion's share of this small heap run
<headius>
ok, 150 is too small too
<headius>
I'll just go with the 250MB heap numbers
<headius>
doing some 100 concurrent runs to see how things change
<headius>
according to system monitor, each Ruby worker is around 110MB resident memory with only about 10MB shared memory
<headius>
so 20x5 is 2GB easy
<headius>
it is handling the load though
<enebo[m]>
headius: are you sure GHA is properly passing password in with MY_PASSWORD?
<headius>
MY_PASSWORD?
<enebo[m]>
It is in ruby.yml
<enebo[m]>
just hard-coding it I think in the file I think it belongs
<headius>
for arjdbc?
<enebo[m]>
yeah I am trying to run against mysql locally
<headius>
it should be
<enebo[m]>
I am printing out config before it tells me I am not using a password and there is no password being passed
<headius>
I iterated on these configs quite a bit before it seemed to actually connect to DB and work
<enebo[m]>
So I think something is not getting hooked up...I guess I should find where the real config is defined
<enebo[m]>
ActiveRecord::JDBCError: Access denied for user 'root'@'172.17.0.1' (using password: NO)
<enebo[m]>
I will poke around I guess
<headius>
G1 uses less memory than parallel at large scale
<headius>
100 thread server settling in around 2.1GB (no -Xmx) on G1 vs 3.4GB on parallel
<headius>
eeeeeeenteresting
<enebo[m]>
hmm I added: password: <%= ENV['MY_PASSWORD'] %>
<enebo[m]>
to test/rails/config.yml and I connected
<headius>
parallel is kicking out more req/s but at what cost
<enebo[m]>
I didn't make a db it expects but that is somethign else
<headius>
enebo: the GHA runs start mysql in a docker container too I believe, so it should be similar for you
<enebo[m]>
yeah so I needed that line to run but it is possible somehow it is provided by another means
<headius>
fwiw I never got it running locally, I just leaned on GHA
<enebo[m]>
./rakelib/db.rake: params['-p'] = ENV['MY_PASSWORD'] if ENV['MY_PASSWORD']
<enebo[m]>
This appears to just be for manually creating/loading mysql locally (without docker)
<enebo[m]>
This is pretty elaborate. IT may work somehow on GHA
<enebo[m]>
I think in travis this just had a local mysql in the image
<enebo[m]>
anyways this is not important today
<headius>
ok looking good to have some show off numbers
<headius>
got some other trip prep to do... I will work from the plane and can update talk up until mid-afternoon tomorrow
<enebo[m]>
ok
<kares[m]>
seeing nice improvement numbers even with the bench_create - indy gets 5-6x faster than non-indy
<kares[m]>
non-indy on 17 is slower than on 11 though - by around 20%
<enebo[m]>
kares: for keywords more IR is getting emitted (on call side if ** is involved)
<enebo[m]>
which is more common in Rails
<enebo[m]>
Ok I am getting some AR tests executing now for mysql
<kares[m]>
interesing ... also an observation leading to smt about argument handling:
<kares[m]>
- on 17 there isn't a difference between create() vs create(some => hash)
<kares[m]>
- create() (no args) is 3x faster than when args are used on Java 11 (no indy)
<enebo[m]>
new method reload_type_map in native connector (I just made it return context.nil for now)
<enebo[m]>
Every receive instr is looking at last arg as hash a little more (pre 9.4) since it might be a keyword hash
<enebo[m]>
Not a lot of logic but some
<enebo[m]>
headius: I would be surprised if mysql isn't already working
<enebo[m]>
I get tons of passing tests but the test abort later on assuming a connection pool which is seemingly not there
<enebo[m]>
but it created the data stuff and was definitely hitting that data
<enebo[m]>
bleh. I think this is another parallel worker issue
<headius>
If I remember right mysql just patched specific methods rather than being a full copy like SQLite, is that correct? If so then there may not be much work to update it since it inherits most of its functionality
<enebo[m]>
They both largely use AR source
<enebo[m]>
SQLite only has one copied file but the connection adapter is almost entirely AR code
<enebo[m]>
mysql mighth ave less but it is similar
<enebo[m]>
postgres is the deviant
<headius>
Well if it looks like it's working share your my sequel docker setup and I'll try to get some numbers on that
<headius>
Mysql
<enebo[m]>
yeah
<headius>
Did you push pre-release gems for all three adapters?
<daveg_lookout[m]>
I'm seeing the following warning in logs after update to 9.3.4.0 + OpenJDK 11:
<daveg_lookout[m]>
Is this of interest and warrant an issue? I can narrow it down if so
<headius>
Yeah that would be good to track down since it will be denied access on recent JDKs
<headius>
Where we need to access stuff we should be explicitly opening them up, or finding another way to accomplish the same goal
<headius>
Since this is using ConstantField it will probably be some Ruby code trying to access that field through our Java integration layer... So it could be a third-party lib that needs to open something or else do it another way
<daveg_lookout[m]>
Makes sense. I don't think I'll get it submitted today, but hopefully tomorrow. Yeah, I suspect it's a third party lib
<enebo[m]>
headius: no. I didn't think they worked at all
<enebo[m]>
fwiw I am not sure how well mysql runs because there are tons of dots but then it crashes thinking there is no pool to get a connection from
<headius>
Ok
<enebo[m]>
but this might work well enough for crud
<headius>
The connection pool code was one of the things I needed to copy into SQLite to get things running
<headius>
I don't think it's changed much but they shuffled some things around
<enebo[m]>
ActiveRecord::ConnectionNotEstablished: No connection pool for 'ActiveRecord::Base' found.
<enebo[m]>
The code which generates that error is checking multiple places too so I am guessing the pool stuff has been migrating around
<enebo[m]>
There is something about shards and roles so maybe not migrating as much as different use cases
<headius>
Hmm
<enebo[m]>
hmm retrieve_connection
<enebo[m]>
who knows if this is normally called or is something special cased
<enebo[m]>
I suppose it means reading source code :|
<kares[m]>
<enebo[m]> "kares: but the TIMES was meant..." <- hmm right, but there's still another block 😉 ... still the numbers were confusing to me - maybe this could be re-done another way
<kares[m]>
<enebo[m]> "HashConfig will freeze what I am..." <- that's new
<kares[m]>
yeah try hooking up before it's about to freeze ... hopefully still possible
<kares[m]>
given all or the xxx_config methods assume a mutable hash atm
<kares[m]>
s/or/of/, s/xxx_config/xxx\_config/
<enebo[m]>
kares: yeah I think it is partially that more happens aound each invocation of measure but nothing on TIMES so it is less overhead. a while would be almost no overhead but it looks weird
<enebo[m]>
ok so I have run into two missing native adapter methods and 3 cases where we have not updated an existing method to have some kwargs on it
<enebo[m]>
Nothing fast in getting mysql up.
<enebo[m]>
Nothing really bad except perhaps the frozen hash and even that might not actually be bad once we start figuring out how our code loads relative to AR code loading
<daveg_lookout[m]>
kares: Unrelated to my comments above, when you have a chance, I'd like understand the test problems that led to https://github.com/jruby/jruby-openssl/commit/201476bc7fbb96f629f8029af7cc188d45608179 . Lookout has an internal gem (originally written by mkristian) that leveraged those methods to hook libgmp into BouncyCasle (fast-rsa-engine, which we once had on our public github, but which was removed since we utterly failed to support
<daveg_lookout[m]>
it). I'd be happy to try and fix the test issues to try and get those methods added back in. Thanks!
<kares[m]>
that was pretty much unused code esp. after Java 9+ where JOSSL stopped hacking the Java security internals
<enebo[m]>
After changing about 5-6 things I can run a migration and a CRUD controller is working
<enebo[m]>
Most of it is just comparing signature changes in AR against what we have in our copies of similar files
<kares[m]>
daveg_lookout: guess we could add smt back but would need to understand the use-case better - you could open a draft PR but just having those commits reverted might not be sufficient
<kares[m]>
it would be best if we knew why the hooks are needed in the first place and whether there's a better way for the "internal" gem to handle this
<daveg_lookout[m]>
Will do. The gem creates new Ciphers and Signatures in Java that leverage libgmp under the hood, then was using the removed methods to add them to the list of available implementations
<daveg_lookout[m]>
Thanks
<kares[m]>
please mention that (as detailed as possible) on the PR ... won't have time to look into this week and I am going to forget everything 😉
<daveg_lookout[m]>
Will do, totally understood
<daveg_lookout[m]>
We also owe a bug report on Sinatra issues with 9.3.4.0. @kroth_lookout ran into this one, I'll bug him to submit it properly
<headius>
On my way to airport and then I will check back in
<headius>
enebo: so no go on mysql crud today?
<enebo[m]>
not without you having to change code
<enebo[m]>
but I can maybe fix a few of these changes today
<enebo[m]>
headius: it is an option for me to send you some instructions with what things I cannot change (e.g. Rails source itself) so you can build and run for numbers perhaps
<headius>
Yeah that would be fine
<enebo[m]>
oh this is odd...this may be less changes than I thought too since I can see bundler hooked into 50.0 for part of this
<headius>
Having my blast off sundae now but laptop is out
<enebo[m]>
which is really odd since I have 70.0pre and it is actually hitting it for some stuff
<headius>
You rebundled?
<enebo[m]>
I thought so. I need to start over here but one specific problem was already fixed in 70.x code
<headius>
we'll see how this laptop does on battery
<headius>
this keyboard has so much travel distance compared to the butterfly nightmare
<headius>
kares: you make a good point about the extra block for the times loop so we can go with your branch
<enebo[m]>
more ruby per block call without it for timing capture but perhaps it doesn't matter
<headius>
we could make it a while loop but we really just need to do better getting rid of the benchmark harness overhead
<headius>
I don't think it will make a difference for this
<headius>
these number with cruby on sqlite3 still seem way too slow
<headius>
1.3k iterations for a boolean field update in 5 seconds
<headius>
versus 88k in JRuby
<enebo[m]>
ActiveRecord::JDBCError: Unknown system variable 'innodb_file_format'
<enebo[m]>
lol
<headius>
the other ones are even worse
<headius>
hah, lovely
<enebo[m]>
I will see what that is but I am on a different stream of errors
<enebo[m]>
if query_value("SELECT @@innodb_file_per_table = 1 AND @@innodb_file_format = 'Barracuda'") == 1
<headius>
hmm... this is actually maxing out a core on CRuby
<headius>
BARRACUDA
<enebo[m]>
In MySQL 8.0, the option innodb_file_format is removed.
<enebo[m]>
good news...only one problem
<enebo[m]>
Even the freeze issue is not there
<headius>
Excellent!
<headius>
If I have an outlet on the plane I can try to get mysql numbers
subbu has joined #jruby
<enebo[m]>
headius: I pushed 70.0.pre of mysql
<enebo[m]>
I commented out a single line about reloading a type map. This needs to be looked at but something changed.
<enebo[m]>
You just need to edit Gemfile and things should work
subbu has quit [Quit: Leaving]
<headius>
Sweet
<enebo[m]>
heh Maybe I will look a postgres tomorrow morning :)
<enebo[m]>
I was planning on auditing recv_keywords tomorrow
<enebo[m]>
The logic might be able to be simpler or not
<headius>
Any extra progress I can report will be great
<headius>
aaaand we're back
<headius>
guess I should have installed docker on the ground
<headius>
4MB/s, not too shabby
<headius>
I'm gonna spend most of the flight just trying to get mysql running
<headius>
woot, benches running
<headius>
hot crackers, this is way faster than sqlite
<headius>
kares: this still seems mega slow on MRI on mysql2, and I notice the warmup phase each bench is slower than the previous one even though they should be similar... any ideas on this?
<headius>
there's definitely someting in the benchmark slowing MRI down
<headius>
enebo: any thoughts?
<headius>
if I remove most of the benchmark cases the perf is a lot higher
<headius>
I don't remember having this problem with the benchmarks before
<headius>
maybe it is adding this all to a transaction and getting slower and slower?
<headius>
I think that was it... MRI is now way faster after I wrapped the updates in a transaction
<headius>
ugh nevermind I was still running JRuby
<headius>
I am stumped
<headius>
it feels like it is development mode perf but I can't figure out how to know or how to force it to production
<enebo[m]>
headius: I think if this is the case you should just break this into n benchmark runs
<enebo[m]>
in theory this is probably a bit better at showing optimization effects anyways since calling the same method at n different sites will warmup differently
<enebo[m]>
with that said I suppose that would also favor us
<enebo[m]>
but better to not see MRI continuously slowing down
<headius>
Maciej Mensfeld on twitter said it could be some problem with fibers and enumerators introduced in 3.x
<headius>
but rvm fails to build 2.7
<headius>
can you try to reproduce this so at least I know I'm not crazy?
<headius>
should just be install rails, install sqlite3, run the bench and it will be way way slower than JRuby
<headius>
on 3.1
<enebo[m]>
Was that branch landed?
<headius>
which branch do you mean
<headius>
IPS?
<enebo[m]>
yeah
<headius>
it is in the main repo, you can merge it
<headius>
I can't use these results so far
<enebo[m]>
yeah I will just merge this
<headius>
ok
<enebo[m]>
how do you specify which db to use
<enebo[m]>
I have not run this in a few years
<headius>
one sec
<enebo[m]>
AR_ADAPTER
<headius>
AR_VERSION="~>7.0" RAILS_ENV=production AR_ADAPTER=sqlite3 rvm ruby-3.1 do ruby bench/benchmark_update.rb