<byteit101[m]>
Ok, I've done a PR for the changes to libfixposix and got a basic binary gem going
<byteit101[m]>
progress
razetime has joined #jruby
<headius>
byteit101: that seems similar to libsass in the sassc gem
<headius>
I'll have a closer look tomorrow if you like but it's probably as good as any way
<byteit101[m]>
Ok, I've implemented something similar
<byteit101[m]>
weird task names, but appears to work at least locally
<byteit101[m]>
that takes care of "normal" gems, but we do need to find a way to package it for bundling jruby which may be different
<headius>
Cool
<byteit101[m]>
I tentatively have a subproject for the jar
<byteit101[m]>
but haven't gotten there yet
<headius>
I'm thinking we'll eventually want something like jnr-nativelib that can be a general solution for shipping, unpacking, and loading prebuilt native libs
<byteit101[m]>
so many moving pieces here, 4 "proper" gems, and 2 other subprojects so far
<headius>
The conversation earlier with ldywicki is relevant here too
<headius>
They are going to help us repackage JFFI and set up the build to do cross-platform binaries
<headius>
So there's probably some collaboration possible here
<byteit101[m]>
I saw it looked interesting, but wasn't sure what it was about exactly
<byteit101[m]>
sound good
<byteit101[m]>
rlimit is strange: On implementations where all resource limits are representable in an object of type rlim_t, RLIM_SAVED_MAX and RLIM_SAVED_CUR need not be distinct from RLIM_INFINITY.
<headius>
Yeah posix is such a good standard
<headius>
I'll be back in the office tomorrow morning
<byteit101[m]>
Ok, umask and rlimit piped through all the gems
<headius>
So I forgot if you mentioned, is the LFP guy willing to reincorporate those changes?
<byteit101[m]>
Ok, if you want to play with this (probably lots of brokenness still) you can at least build it via `rake dev build` in the top repo now
<byteit101[m]>
time for bed now though
<byteit101[m]>
then `require 'subspawn/binary'; require 'subspawn'` and you have access to SubSpawn. Add `require 'subspawn/replace'` and PTY and Process are upgraded
razetime has quit [Ping timeout: 246 seconds]
razetime has joined #jruby
sagax has quit [Remote host closed the connection]
razetime[m] has quit [Quit: Bridge terminating on SIGTERM]
enebo[m] has quit [Quit: Bridge terminating on SIGTERM]
brometeo[m] has quit [Quit: Bridge terminating on SIGTERM]
kares[m] has quit [Quit: Bridge terminating on SIGTERM]
headius has quit [Quit: Bridge terminating on SIGTERM]
byteit101[m] has quit [Quit: Bridge terminating on SIGTERM]
PierreJacoboni[m has quit [Quit: Bridge terminating on SIGTERM]
makePar[m] has quit [Quit: Bridge terminating on SIGTERM]
ahorek[m] has quit [Quit: Bridge terminating on SIGTERM]
ldywicki[m] has quit [Quit: Bridge terminating on SIGTERM]
nilsding has quit [Quit: Bridge terminating on SIGTERM]
lopex[m] has quit [Quit: Bridge terminating on SIGTERM]
enebo[m] has joined #jruby
razetime[m] has joined #jruby
PierreJacoboni[m has joined #jruby
makePar[m] has joined #jruby
byteit101[m] has joined #jruby
ahorek[m] has joined #jruby
ldywicki[m] has joined #jruby
brometeo[m] has joined #jruby
kares[m] has joined #jruby
nilsding has joined #jruby
headius has joined #jruby
lopex[m] has joined #jruby
sagax has joined #jruby
Guimauve has joined #jruby
Guimauve has quit [Client Quit]
razetime has quit [Ping timeout: 252 seconds]
ldywicki[m] has quit [*.net *.split]
ldywicki[m] has joined #jruby
sagax has quit [Remote host closed the connection]
subbu has joined #jruby
subbu has quit [Quit: Leaving]
<headius>
enebo: I'm trying to update rdoc to 6.4.0 in jruby-9.3 but when I do so it breaks build trying to install psych
<headius>
maybe you can give it a try and confirm whether it's just me?
<headius>
I'll try pushing as a PR and see what happens
<enebo[m]>
headius: I will give it a shot
<headius>
it looks like it failed in CI too
<headius>
something about this rdoc update causes the psych install to fail
<headius>
oh I see
<headius>
it pulls in psych 4
<headius>
our build system doesn't handle conflicting dependencies well so it just craps out
<headius>
the previous rdoc does not depend on psych so I will try that and confirm it still fixes the CVEs we are trying to fix
<enebo[m]>
you already figured it out but yeah it complains about no psych-3.3.3
<headius>
ok same thing
<enebo[m]>
So today and tomorrow (although some hope for today yet) I should have combined parser good enough to merge
<headius>
psych 4 might be fine but I don't want to update more than necessary
<headius>
oh exciting!
<enebo[m]>
I was at 25 F/E on Monday 27 F/E on Tuesday and I am 26 F/E this morning :)
<headius>
split the difference
<enebo[m]>
It has been very much a 2 steps forwards 2 strap back for last couple of weeks but each time the thing causing the shift is less and less important
<headius>
the rdoc 6.4.0 commit set is weird and seems to include all of the previous commits even if I compare with 6.3.3, but it seems like the important parts are all in 6.3.3
<enebo[m]>
I have also fixed a few things which I am sure never worked in past versions so that is also nice
subbu has joined #jruby
<headius>
I will try that and confirm with reporter that this is good enough
<headius>
6.3.3. builds fine
<enebo[m]>
cool
<headius>
enebo: that woman doing the other JRuby talk, Yarden, might stop by to chat about how our bytecode looks
<headius>
one problem she ran into hooking up our bytecode to a JVM debugger is due to how we encode the variable names ("a" becomes $t_a_4 for example) and the fact that sometimes they are in DynamicScope
<headius>
I think we can make some simple improvements to make the bytecode more debuggable
razetime has joined #jruby
yardenlaif[m] has joined #jruby
<yardenlaif[m]>
Hey! Thanks for the introduction headius !
<headius>
Hi there!
<yardenlaif[m]>
So actually with $t_a_4 the parsing was simple, especially since I could get all the variable names from the $staticScope variable
<headius>
yeah it is just something you have to know or guess right... there's really no reason we couldn't make the real local variable slots just be the variable name
<headius>
then you would just grab the unprefixed names for the normal local variables... which would cover all cases without a dynamic scope
<yardenlaif[m]>
I was actually worried there might be more than one temp variable per real Ruby variable (aka $t_a_4, $t_a_5...)
<headius>
I don't *think* so but enebo may know more
<headius>
I think the number is just a monotonic index of the variable
<headius>
so it's "a" but it's also the 4th variable we encountered during compilation
<yardenlaif[m]>
I see
<headius>
so a method that has just "a" and some calls looks like this for me locally:
<yardenlaif[m]>
In this case I guess `hi` is opted out
<headius>
yes
<headius>
dead code
<yardenlaif[m]>
Though staticScope does show me there is one variable called `hi` in scope
<headius>
for debugging it would be possible for us to disable such optimizations, if that is helpful
<headius>
yeah we still need it for metaprogramming, like `local_variables` call
<headius>
it just isn't used so we don't emit it into the bytecode
<headius>
and don't assign it and don't create the string, etc
<yardenlaif[m]>
I wouldn't want to hurt optimizations like this in my case
<headius>
right, just saying if you wanted it in the debugger it can be done
<headius>
line numbers etc should also still make sense and it would just skip the dead line
<yardenlaif[m]>
Yes they do make sense
<headius>
this is really exciting, I wasn't just being hyperbolic... I have only ever made this work in jdb because of the way IDE debuggers set up source paths
<headius>
but I always knew we could make it work right
<yardenlaif[m]>
How do source paths help?
<headius>
IntelliJ for example only adds the source paths set up in the project, and only as relative paths, for the debugger to find them
<headius>
we emit the bytecode with an absolute path
<yardenlaif[m]>
I don't think a standard java debugger needs a lot of tweaks in order to support JRuby debugging
<headius>
so that could be improved so that IntelliJ could automatically see the Ruby sources
<yardenlaif[m]>
oh, our debugger doesn't work that way so I didn't have that problem :)
<headius>
yeah it does just work but to show the source lines it needs to be able to find the sources
<headius>
nice, what debugger is that?
<headius>
so like I can start a debugging session in IntelliJ but when it gets to the JRuby bytecode it just skips over it
<headius>
because it doesn't know what sources to step into
<yardenlaif[m]>
It's a debugger for production, and it's a SaaS program so we don't have access to our code
<yardenlaif[m]>
Ohhh I see
<yardenlaif[m]>
Yeah we don't have those sorts of features because of our environment
<yardenlaif[m]>
*to the code being debugged, not to our code
<headius>
and I don't really want to add / to the list of sources 😄
<headius>
ok interesting
<yardenlaif[m]>
Basically it just knows how to place a breakpoint on file:line, and I'm guessing intellij can do that as well
<headius>
in any case improvements we could make on JRuby side should help all debuggers, like giving normal Ruby local vars better names
<headius>
it can but it need to know that <project root>/lib/source.rb is actually <full path to project root>/lib/source.rb
<headius>
or we need to put the relative path in the bytecode's filename attribute
<headius>
other IDEs might be better at this, I don't know
<yardenlaif[m]>
I would expect intellij to be able to handle relative paths, but I guess they just don't need to?
<headius>
right... Java and most other JVM languages obey Java package structure, so they always embed the relative path, so it will map properly to the project's relative paths
<headius>
we use full path because Ruby does not have any strict pathing structure
<headius>
in Ruby you might load foo/bar.rb from two places depending on LOAD_PATH manipulation
<headius>
which is not possible in Java (other than classloader tricks, which would also confound an IDE)
<headius>
I have to step away briefly but if enebo is around he can answer more questions
<headius>
enebo: the background on this is that yardenlaif is doing a talk on her company's JRuby experiences at RubyConfTH and part of that is how they hooked up their JVM debugger to JRuby bytecode
<enebo[m]>
I was out in the great outdoors but I am back...although I will be eating lunch soon
<enebo[m]>
I will read back a bit
<enebo[m]>
yardenlaif: all temps are just from a single variable so it does not matter on the temp variable type. This used to be different where each temp maintained its own counter (and used to be stored on IRScope instead of IRBuilder) but I moved to a single counter as part of moving it
<enebo[m]>
There was no specific reason for either choice though.
<headius>
I think a nice change would be to decode simple object local variables to their simple Ruby name as a start
<headius>
For purposes of debugging Ruby code, rather than debugging our compiler, it might also make sense to omit the mangled variable names
<headius>
It's helpful to me as the jit maintainer, but it's unnecessary visual noise for anyone only interested in the Ruby attributes
<byteit101[m]>
yardenlaif: It's not really relevant to what I think you are doing, but some of the most, uh, "unique" bytecode is in the reified (aka become_java!) class boilerplate
<enebo[m]>
if so then we should also make it simple to see shadowed names somehow as well
<headius>
Shadowed names?
<headius>
byteit101: thank goodness the debugger would skip over that code, at least the parts that aren't generated with source files and line numbers
<enebo[m]>
proc { |a| proc { |a| } }
<headius>
Oh yes
<headius>
Those cases may not show up in the byte code though since the local variables are punted to the heap
<headius>
That innermost a would probably show up in the bike code but the outer one probably would not
<headius>
Heh bytecode
<headius>
I think the local variable change would be pretty simple so I might just go ahead with that. Once we learn more about yardenlaif debugger We can consider other improvements
<headius>
I'd really like to fix that source file thing so that IDEs could automatically pick up the Ruby sources
<headius>
enebo: did we ever get anywhere with chunking large methods?
<headius>
It seems like Rubyists have mostly stopped creating huge methods, but generated code frequently still breaks the bank
<enebo[m]>
JVMVisitor needed a bunch of changes. I had started on changing it but there was something confusing between scopes and the method itself
<byteit101[m]>
I do remember annotating a few bits with source/line numbers when it made backtraces more reasonable in failures
<enebo[m]>
This was years ago but I think it was something like I expected some top-level thing for emitting the scope but the top layer is sort of doing two things. I just don't really remember
<byteit101[m]>
oh headius, If you recall I made several comments of "both is best" regarding https://github.com/ruby/irb/pull/403 as while the irb solution is complete, the jruby solution isn't. Though jrub+pry is currently complete
<headius>
Oh, so it's still possible to crash it if you create a module that doesn't respond to the necessary methods, which is not specific to JRuby
<headius>
I understand
subbu has quit [Quit: Leaving]
<byteit101[m]>
How should I set up jruby builtin/jar packaging of subspawn? I think I shall attempt that this evening
<ldywicki[m]>
hm... one question, as I always thought that jnr works without native parts (by some magic), was I wrong and it is using libffi just like jna?
<headius>
ldywicki: it is using libffi built into the jffi JNI extension, yes
<headius>
correct
<ldywicki[m]>
so that's why you do native extraction
<ldywicki[m]>
so now that I understand core, what is the role of jnr-posix, jnr-a64asm ?
<ldywicki[m]>
I am actually curious of possibility to use jnr to write down a handler for low level linux socket through ie. linux kernel xdp and related protocol family. So far most of raw socket handlers I found for java which are below tcp/ip at mac/ether layer ship jni wrappers. JavaCAN is one example, but same thing applies to libraries which attempt to send ICMP and other packets.
<headius>
jnr-posix is a set of pre-bound POSIX functions, mostly driven by what we need in JRuby
<headius>
jnr-a64asm and jnr-x86asm are used to generate assembly coded stubs for simple function calls in order to skip some JNI and libffi overhead
<headius>
a64 = aarch64
<headius>
you'll also encounter jnr-enxio, which is an NIO implementation atop POSIX IO functions and selection (epoll, kqueue), jnr-unixsocket which is what it sounds like, jnr-process which is pregenerated POSIX constants like O_RDWR across platforms, and jnr-process which is a java.lang.ProcessBuilder/Process look-alike based on posix_spawn
<headius>
the core FFI functionality is provided by jffi (the native bits) and jnr-ffi (the API for loading and binding libraries
<ldywicki[m]>
I was mistaken by my own assumptions cause I thought that all work JNR does is provided by specific asm modules, I haven't noted the libffi part for some reason. I made an attempt with raw sockets based on unixsocket, which I failed two years ago. I might be re-trying it soon. :)
<ldywicki[m]>
Now I have clear picture, I think I will be able to have a look on poms and dependencies tomorrow. I have couple of calls around noon, but then I should have a spare hour or so to start with cleanups.
<headius>
ahh I see
<headius>
yeah it would be neat if we generated ALL of the native code at runtime, but that would be a big project 😄
<headius>
I'll b
<headius>
I'll be in the office tomorrow also
<headius>
enebo: I think I know roughly the problem with this Time.at thing
<headius>
it was logic duplicated from CRuby and I think I probably just mapped it to joda incorrectly
<headius>
it is adding the new zone on top of the current zone so in -0400 it looks like it is doubling, but when I run it locally it adds -0400 and local -0500 and produces -0900
<headius>
so it is not basing the adjusted time off UTC
<byteit101[m]>
headius: any pointers on how should I set up jruby builtin/jar packaging of subspawn?
<headius>
Like for internal uses of spawn that won't load gems?
<byteit101[m]>
yes
<byteit101[m]>
platform=java is slightly different, but I know how I can do that with a normal rvm install. embedded/bundled I'm not so sure about
<byteit101[m]>
Actually, is there a way to have like java-x86-linux or such gems? ie. ignore the fact that we are on java for the gem downlaod?
<headius>
Yeah this is where the challenge of shipping cross-platform made of binaries other than JFFI comes into play
<headius>
The short answer is there is no answer right now
<byteit101[m]>
would adding ffi-$host as a rubygems target be a reasonable feature request?
<headius>
We could ship what you have as Ruby code and hope that the library is already present on the host system, but we don't have a good way of distributing the binary
<byteit101[m]>
alas that is less helpful now that I changed libfixposix for the ulimit and rlimit
<byteit101[m]>
*umask
<headius>
The only native binary we distribute as part of the jar is the JFFI stub
<headius>
Well even without those we still have chdir
<headius>
If the library is available
<headius>
chdir support would fix a whole bunch of issues
<byteit101[m]>
not installed by default on ubuntu/debian, so without that you have nothing :-)
<headius>
Yep that is true
<headius>
This is why I'm hoping that the work on doing a better job of packaging the JFFI native binary will provide a path forward to ship other native binaries including LFP
<headius>
This is also why I considered the gross solution of including LFP in the JFFI stub
<headius>
Because that binary we already handle
<headius>
There is a maven plug-in for building and shipping cross-platform native binaries, including the unpacking and loading, but I've only read through the docs a little bit
<headius>
So we are in unfamiliar territory
<headius>
JFFI itself is a special case because it has its own unpacking and loading logic that is not generic
<byteit101[m]>
> would adding ffi-$host as a rubygems target be a reasonable feature request?
<byteit101[m]>
^ ?
<headius>
It is an interesting idea
<headius>
I guess I don't know the answer
<headius>
We have proposed enhancing the platform logic in the past but it is very much tied into the structure of the gem file name and we can't introduce ambiguities there
<headius>
In any case it would not be a short-term enhancement
<headius>
The way we handle FFI gems right now is to either have them build the library on install or to ship all platform binaries in the gem itself
<headius>
Or assume the library is on the system already of course
<headius>
There's also been some recent discussion about enhancing gym dependencies to know about platform libraries that must be installed but again that is just an early discussion
<headius>
Sure, open an issue there. There's a good chance I've opened an issue in the past related to this too
<byteit101[m]>
> There is a maven plug-in for building and shipping cross-platform native binaries, including the unpacking and loading, but I've only read through the docs a little bit
<byteit101[m]>
do you know the plugin name?
<byteit101[m]>
What does truffle ruby have as the platform?
<headius>
maven-native-something
<byteit101[m]>
:-D
<headius>
Because they support C extensions they match what CRuby reports
<headius>
They don't have any TR specific extension API
<byteit101[m]>
ok hmm
<headius>
But it might make sense for all implementations to have a way to specify FFI dependencies that ship a native binary
<byteit101[m]>
jruby doesn't support building "c extensions" that are just native code, right?
<byteit101[m]>
(gem install time build)
<headius>
Yes that works fine. That's how the sassc gems works
<headius>
As long as the appropriate make tools are available on the local system and the gem is just a plain native library, it can be built on install
<headius>
That would work fine for this case except that if we ship the gem in the box there's no install time at which to build the library
<headius>
So that comes back to shipping with degraded spawn features and recommending people install the gem to get upgraded features
<headius>
There are many options and they're all a little bit hacky
<byteit101[m]>
who should I cc on this proposal, you, eregon, and anyone else?