<byteit101[m]>
headius: current idea is we just duplicate jffi Library.getCachedInstance and openLibrary to take a URL, then duplicate the logic in StubLoader to extract the files, then call the original getCachedInstance/openLibrary with the extracted files. JRuby will then call JRubyFile.createResource to decode all supported uris to pass to jffi
yosafbridge has quit [Ping timeout: 252 seconds]
yosafbridge has joined #jruby
razetime has joined #jruby
razetime has quit [Ping timeout: 255 seconds]
subbu has joined #jruby
subbu has quit [Quit: Leaving]
razetime has joined #jruby
<ldywicki[m]>
just got first native compilation passing for libtest ;-)
<ldywicki[m]>
trying to get a cross compile for several platforms to see how it goes
subbu has joined #jruby
subbu has quit [Ping timeout: 252 seconds]
subbu has joined #jruby
<ldywicki[m]>
got libtest checked with x86_32/64, armv5/v6/v7/v7a/v7l, aarch64, mips, mipsel, ppc64le, risc 32/64, looking how it will go with windows x84_32/64, windows armv7 and arm64
<ldywicki[m]>
its mainly based on javacan pom.xml sample and compiler/linker script they got (bypassing make). I see how it gets wired in.
<ldywicki[m]>
Once I have libtest (it has no dependencies) covered under windows I will be able to replicate same procedure to libffi and jffi
Freaky has quit [Quit: reboot]
Freaky has joined #jruby
<ldywicki[m]>
headius: ☝️
<headius>
Hey there
<headius>
Sick kid, late morning but I'm in the office now
<byteit101[m]>
headius: ^ (last night's message) is that an ok way to go about it?
<headius>
catching up now
<headius>
byteit101: that might be a fine way to integrate it into JRuby but I would like to see a standalone library that can handle selecting the platform and unpacking libraries
<headius>
something like those maven plugins/libraries or just those if we can make them work like we want
<headius>
ldywicki: sounds like great progress... at least we will be able to get cross-platform builds for linux completely automated
<byteit101[m]>
unfortunately that's hard to do, as ffi/libray.rb does the path searching and sends jruby core the absolute path
<ldywicki[m]>
headius: I think I will be able to get windows dll out of dockcross as well, just need to figure out necessary mingw options
<byteit101[m]>
require 'a_c_ext' is a different codepath entirely, but in theory would do the platform searching itself as you would like
<ldywicki[m]>
osx native build asked is another challenge, will look at it after windows. The signing stuff required by OpenRefine is not a blocker, if you have a chance to get this into pipeline then I think I will be able to integrate that
<ldywicki[m]>
one problem I found - C sources are currently not compatible with c99, not sure if that's an issue or intentional violation ;-)
razetime has quit [Remote host closed the connection]
<headius>
byteit101: so I figured this would need to be a callback-based library, where you register a piece of code that can retrieve the native library resource and return a stream of it to the unpacking library
<headius>
the problem with just having a separate library is that Java modules do not allow modules to see other modules' classloader resources
<headius>
so some of this logic has to live within the jar/module we load that has the library
<byteit101[m]>
That's not an issue for JRuby though, right?
<headius>
if we ship prebuilt libfixposix within jruby's main module, no
<headius>
the maven plugin does not look like what we need
<headius>
now that I poke around the code a bit... it just unpacks native libs included in the project as part of the maven build
<byteit101[m]>
Yes, I saw that, and jnr/jffi StubLoader does that already
<byteit101[m]>
do you know why jnr/jffi StubLoader is only called reflectively?
<headius>
I saw you ask that... no, I'm not sure
<headius>
usually that would be to avoid loading more than necessary at boot but there's not much downstream from StubLoader
<byteit101[m]>
it looked classpath related somehow
<byteit101[m]>
*classloader
<headius>
hmmm could be trying to localize the library loading to a single classloader so we don't end up loading twice
<byteit101[m]>
at any rate, I guess I'm not understanding what n-l-l or such would get us? granted I'm looking at this only from a ruby perspecive
<byteit101[m]>
all ruby FFI gems must deal with the platform themselves today
<byteit101[m]>
unless I'm missing something?
<headius>
ah well let's clarify what we are trying to solve for now then... you are currently just trying to figure out how to handle the native libraries in the gem?
<headius>
it wasn't hard and might be a good way for us to bootstrap some other-platform builds until we have cross compiling set up
subbu has quit [Quit: Leaving]
<byteit101[m]>
> you are currently just trying to figure out how to handle the native libraries in the gem?
<byteit101[m]>
Yes
<byteit101[m]>
I can do more if necessary, but I would like to keep the scope small as this is already a project touching lots of areas
<headius>
yeah I understand
<headius>
have you looked into what the sassc gem does, building the library on install
<headius>
shipping prebuilt libraries is fine but as you say it's then up to you to select the right one and load it
<byteit101[m]>
yes, was going to do that for published gems, but we can't do that for builtin jems in the complete jar
<byteit101[m]>
rvm it may be fine though
<headius>
ok so then we are also talking about native libraries in the gem inside complete jar
<headius>
just trying to compartmentalize what we need to solve right now
<byteit101[m]>
correct
<headius>
right so the little code snippit you showed the other night, adding some logic into the FFI loader to unpack stuff coming from classloader resources... that seems like a fine start
<headius>
I feel like we have been going around and around but a simple solution for now can become a more complete solution later
<headius>
we just need to get the libraries out of the classloader
<headius>
if the logic in the gem is already doing the selection of a native lib than we just need unpacking before loading, which your patch provided
<byteit101[m]>
yes, the "simplest" from the perspective of the least parties involed is what I have on my branch in just jruby, but I know you didn't like that
<headius>
I mostly wanted to see if there was something else we could use that would handle more of the sharp edges for us, but what you have seems ok to put into a PR and iterate on
<byteit101[m]>
You want the second option here, yes? {jruby-only (on branch), jrub+jnr (snippet from last night)}
<headius>
I don't see a snippit from last night, just your musing on how to yank out the logic from JNR and use it in JRuby for this
<headius>
I guess I should just say I don't know what's best... if what you came up with for jruby-only doesn't gain much using the jnr logic then it doesn't matter much
<headius>
if the jnr logic is useful and can be generified a little bit and used for this, then that saves us creating new code
<byteit101[m]>
well, that brings us back to the question of if the jnr logic can be used anyway, ie: do you know why jnr/jffi StubLoader is only called reflectively?
<headius>
so let's skip the discussion about making a one-library-to-rule-them-all for all loading and just evaluate the simple options we have for this as a one-off problem
<headius>
yeah so it is an attempt to force library loads through system classloader... this is probably to avoid double-loading the same library in multiple apps
<headius>
the jni stub is loaded by the JVM and managed at a classloader level
<headius>
our dlload'ed libraries don't fall into that category since we're just going straight to the metal
<byteit101[m]>
the question thus is: can I refactor the logic out of that class?
<byteit101[m]>
(ie outbound calls)
<byteit101[m]>
(extraction logic)
subbu has quit [Quit: Leaving]
<headius>
hmm yeah this logic at least is fairly jni-specific and not applicable to generic library loads by FFI
<headius>
StubLibrary does have all the logic for platform selection and unpacking and such but it would need modifications to be used with any old library
<byteit101[m]>
that's what I noticed: everthing JVM is for JNI libraries, which are a bit different than true native libraries
<headius>
most of the selection logic would be done by the gem or FFI though in your case
<headius>
sassc does have some logic to select prebuilt libraries but I have not looked into that... mostly to cover a similar case where you can't always build a library: Windows
<headius>
so it will build the library at install time if you can't install a prebuilt version
<headius>
the prebuilt combined with your simpler FFI patch seems like it's fine for now since it isn't duplicating very much of the jffi logic
<byteit101[m]>
Ok, so no changes to my branch then? do you want a separate PR for just that change? or should I have it as it is now, in the pty.spawn PR?
<headius>
let's do it separate and we can review and iterate on it first
<byteit101[m]>
k
<headius>
do you have a published gem with multiple prebuilt libs right now?
<headius>
just thinking about this from a testing perspective
<byteit101[m]>
No, I was wainting for your help in building aarch64
<headius>
ok well I'm waiting for my ssh key to propagate on this gcc compile farm and then we have many platforms available
<byteit101[m]>
oh another reason I have not yet done that: what version should I use? 0.4.3 was the last release, 0.5.0 is the next release in the source code, but my changes haven't been merged by sionescu. What should I do? 0.4.8 or 0.4.9?
<headius>
hmm
<headius>
I would ping him again to see if a release is coming soon, and if not and we want to get this our there immediately we just do a soft fork for now and release with that
<byteit101[m]>
a release isn't good if he doesn't merge my changes first ;-D
<byteit101[m]>
but can do
<headius>
right
<headius>
I mean release + your changes
<byteit101[m]>
poked
<headius>
ok
<headius>
where did you poke? I could weigh in from JRuby perspective
<headius>
enebo: so it turns out the excludes feature as added to MRI test suite does support multiple paths
<headius>
I'm going to make an excludes_wip so we can have a green run of all MRI suites but then do a separate run without those excludes to show the WIP like with specs
<headius>
we could extend this to support "includes" and only run certain tests... that would be nice
<headius>
mspec tags are still more efficient but it would get us parity
_whitelogger has joined #jruby
<headius>
woot this seems to be working
<headius>
I even wrote a script to scrape the rake test:mri output and create excludes
<headius>
enebo: I will push a PR for this... it will add a second "excludes_wip" dir that is used during the main builds (which should be green) and then some additional "wip" rake mri test targets to run without those excludes
<headius>
so we can have the main suites be green and pass the build if we don't regress while still reporting WIP failures as a non-critical CI build
<byteit101[m]>
cool, thanks1
<byteit101[m]>
I think I'm going to try to get ci building a fat gem tonight