<headius>
Tomorrow will work... End of week got busy with other things for me
kares[m] has joined #jruby
<kares[m]>
fyi: have just setup thans.dev on a number of repos under jruby - jruby-openssl, ar-jdbc, jruby-rack
<kares[m]>
as a naive effort to try direct some funds into these ... if there are any objections I am fine to take it down
<enebo[m]>
I will look at it a bit more but I do not have an issue with this unless somehow it ends up involving my taxes
<headius>
I got pinged by them as well. I'm fine with it but we will need to figure out tax ramifications if we start accepting donations
<kares[m]>
I am not 100% sure how it works ... they mentioned it would be nice to have the setup
<kares[m]>
and that sentry uses JRuby, I did not know ...
razetime[m] has quit [Quit: Bridge terminating on SIGTERM]
enebo[m] has quit [Quit: Bridge terminating on SIGTERM]
nilsding has quit [Quit: Bridge terminating on SIGTERM]
yardenlaif[m] has quit [Quit: Bridge terminating on SIGTERM]
ldywicki[m] has quit [Quit: Bridge terminating on SIGTERM]
makePar[m] has quit [Quit: Bridge terminating on SIGTERM]
headius has quit [Quit: Bridge terminating on SIGTERM]
lopex[m] has quit [Quit: Bridge terminating on SIGTERM]
ahorek[m] has quit [Quit: Bridge terminating on SIGTERM]
byteit101[m] has quit [Quit: Bridge terminating on SIGTERM]
PierreJacoboni[m has quit [Quit: Bridge terminating on SIGTERM]
kares[m] has quit [Quit: Bridge terminating on SIGTERM]
headius has joined #jruby
<headius>
maybe they can fund some ossl work 😄
enebo[m] has joined #jruby
razetime[m] has joined #jruby
PierreJacoboni[m has joined #jruby
yardenlaif[m] has joined #jruby
makePar[m] has joined #jruby
kares[m] has joined #jruby
ahorek[m] has joined #jruby
ldywicki[m] has joined #jruby
lopex[m] has joined #jruby
nilsding has joined #jruby
byteit101[m] has joined #jruby
<kares[m]>
maybe, although I have no idea how much popular sentry is these days
<kares[m]>
maybe, although I have no idea how much popular sentry is these days
subbu has joined #jruby
<headius>
darn, so close to having loongarch64 build jffi in CI
<headius>
I need to figure out how to get the most recent qemu now to get platform support
<headius>
byteit101: I am out for lunch for a while, back later
<byteit101[m]>
Ok
subbu has quit [Ping timeout: 272 seconds]
<byteit101[m]>
headius: let me know when you have some time. I'm free the rest of the day
<headius>
byteit101: ok I'm back in the office... Where should we start
<byteit101[m]>
Several good options: native library extracting/loading, building, API review, gem split review
<byteit101[m]>
(pick one)
<headius>
API might be a good one to knock down, what is there to the API above and beyond the standard spawn API from Ruby?
subbu has joined #jruby
subbu has quit [Changing host]
subbu has joined #jruby
<byteit101[m]>
So I made some modifications to the high level API that mirrors standard. There are several layers to this too: Mid-level is closer to posix, and doesn't do parsing, etc. The mid-level API review is sort of part of the gem split review
<byteit101[m]>
In the repo, replace-pty.rb has the PTY API, subspawn.rb has the Process.spawn api, and posix.rb has the mid-level API
<byteit101[m]>
I also made an rbs file for the mid-level (I think I'm missing one or two methods right now)
<byteit101[m]>
The Process.spawn api is SubSpawn.spawn_compat, whereas SubSpawn.spawn is a slight modification to a that API
<byteit101[m]>
however both share most of the parsing logic
<byteit101[m]>
which I did change
<byteit101[m]>
ex: Process.spawn now accepts :pty arguments when using replace
<byteit101[m]>
making subspawn pass process.spawn tests caused a lot of weirdness (to_ary, to_str, to_hash, etc)
<byteit101[m]>
I considered adding multiple extra values to :in (:stdin, :input, etc), and other keys
<byteit101[m]>
as pgroup has multiple already
<byteit101[m]>
:env is new, as is some of the uid, gid, keys
<byteit101[m]>
everything in the switch in subspawn.rb that has a P.s comment is in ruby Process.spawn, everything else I added
<byteit101[m]>
that's the high-level api review
<byteit101[m]>
the mid-level is essentially: how smart or dumb should I make it? It's relatively close to libfixposix calls, though deviates slightly
<byteit101[m]>
Another API decision: which types of errors, errors vs warn, etc. I added a lot of argument errors for ::spawn specs, but lots is untested
<byteit101[m]>
(also, I notice that PTY isn't in rubyspecs)
genpaku has quit [Remote host closed the connection]
<byteit101[m]>
I also wasn't entirely sure how to do signal_mask and signal_default in the high-level API
<byteit101[m]>
oh, and SubSpawn.spawn is there in case you (essentially MRI users or win32 JRuby users) don't want to replace Process.spawn.
<headius>
ok I'm looking at it
<byteit101[m]>
oh and you will need help building it if you want to run it
<byteit101[m]>
It may work as-is, but the build process isn't smooth yet
genpaku has joined #jruby
<byteit101[m]>
high-level is mostly hash-based API, whereas mid-level is builder based
<headius>
ffi-generator is a new creation? There was an ffi_gen a long time ago that never went anywhere
<byteit101[m]>
Yes
<byteit101[m]>
run it via `rake dev` in the parent (checkout) dir
<byteit101[m]>
yea, I was looking for something like that but couldn't find anything
<byteit101[m]>
Found some crystal and python stuff
<byteit101[m]>
ffi_gen is SPECIFICALLY for this code base, it only has enough edge cases for this
<byteit101[m]>
it's not packaged
<byteit101[m]>
just a dev "dependency"
<headius>
so cast is an ext for CRuby but only used to regenerate the .rb binding
<byteit101[m]>
correct
<byteit101[m]>
C AST
<byteit101[m]>
you can do rake dev in cruby, then the rest of stuff in jruby
<byteit101[m]>
We can commit that generated file if you'd prefer
<headius>
Subspawn::POSIX is the minimal fluent wrapper around the FFI binding I see
<byteit101[m]>
Correct.
<headius>
that all looks fine
<byteit101[m]>
Subspawn.rb then aliases it to SubSpawn::Platform = SubSpawn::POSIX
<headius>
ok
<headius>
yup I see that now
<byteit101[m]>
pty.rb + replace-pty.rb make a complete replacement for the builtin pty.rb
<headius>
sad how much easier it is to understand the argument processing in Ruby than in Java or C versions
<byteit101[m]>
Ohgo d yes
<headius>
I think an early version of that logic I actually did some pseudo-ruby just to try to understand it
<headius>
the C code is so confusing
<byteit101[m]>
I would spend 20 minutes to figure out what Cruby was doing over 100 lines, then write 2 replacement ruby lines
<headius>
it is also doing a lot of defensive copying expecting you may fork and then parent and child have to react differently
<headius>
annoying
<headius>
ok and so you require subspawn/replace-pty or replace-builtin to patch over the existing spawns
<byteit101[m]>
correct
<byteit101[m]>
*incorrect
<byteit101[m]>
users: require subspawn/replace
<headius>
this all seems fine to me
<byteit101[m]>
jruby: require subspawn/replace-*
<byteit101[m]>
or if you just want to use subspawn without replacement, just require subspawn
<headius>
enebo might have some thoughts on the API but this layout and split seems ok
<byteit101[m]>
or use one of the lower level api's with 'subspawn/posix' or 'libfixposix'
<headius>
it is a lot of gems for perhaps unknown value in the future
<headius>
I think it is probably better than trying to put too much in one gem
<byteit101[m]>
I mean I'm working on my multi-headed embedded debugger and it's handy
<headius>
yeah it is a logical split, just maybe we'll find that YAGNI
<byteit101[m]>
yes, that was why I wanted it to be reviewed, as 4 gems seems a lot :-)
<headius>
I don't have a strong opinion there... it's yours to maintain and if this makes it cleaner it seems fine
<byteit101[m]>
the lfp gems could be combined, but that precludes using the system lfp or compiling it yourself externally
<enebo[m]>
I don't have anything to note on this
<headius>
so then... API at each level seems logical and straightforward, replacement procedure seems good
<byteit101[m]>
lfp also has features aside from spawn, so hence that's its own gem
<headius>
right that also makes sense
<byteit101[m]>
The temporary generation logic of the fd mapping is completely different. I got confused at MRI and just wrote a brand-new graph-based one
<byteit101[m]>
A key question I do have about the API is how many keys should we accept? and what should we use for the new ones:
<byteit101[m]>
when :uid, :userid, :user, :owner, :ownerid # TODO: which?
<headius>
what's the sequence to build and test everything
<headius>
the base API should just support what CRuby does, and beyond that I don't have a strong opinion on other names and aliases
<byteit101[m]>
build: (once) rake dev, then package everything with rake build at root or in a specific folder, though ffi-binary has lots of targets (binary, local, etc)
<byteit101[m]>
testing: only subspawn and subspawn-posix have tests, though I've mostly been using io/console and rubyspec tests (external) recently
<byteit101[m]>
base API is a strict superset, as I was hoping to "enhance" Open3, etc via new kw args
<headius>
$ rake dev
<headius>
rake aborted!
<headius>
cd libfixposix/src/include/lfp
<headius>
Errno::ENOENT: No such file or directory @ dir_chdir - libfixposix/src/include/lfp
<byteit101[m]>
git submodule
<headius>
oh right just noticed that was a sub
<byteit101[m]>
that submodule currently points to my fork as my PR is still pending
<byteit101[m]>
for fun and confusion, look at fake.h (the one with more than just an include)
<byteit101[m]>
you must be in ffi-binary for that to work
<byteit101[m]>
*the args
<headius>
rake build still tries to build ffi-binary and blows up, not sure how to pass that through
<headius>
but it did work in ffi-binary dir
<byteit101[m]>
comment it out for now :-)
<byteit101[m]>
but that's good
<byteit101[m]>
just do each of the other folders manually
<byteit101[m]>
oh, hmmm
<byteit101[m]>
also I don't have an install task on all of the folders, so if you want to install (vs using -I) you will need to gem install them manually
<headius>
byteit101: commenting out both ffi-binary sections from rake build looks like it built everything else fine
<headius>
oh heh
<headius>
cd ffi-bindings-libfixposix
<headius>
rake build
<headius>
ffi-bindings-libfixposix 0.BINARY.pre.NOT.pre.BUILT.pre.ERROR built to pkg/ffi-bindings-libfixposix-0.BINARY.pre.NOT.pre.BUILT.pre.ERROR.gem.
<byteit101[m]>
^ try the edit i suggested above
<byteit101[m]>
oh yeah, i couldnt figure out how to make normal looking versions
<headius>
heheh
<headius>
yeah that's pretty not normal
<headius>
your edit above worked for rake build at root
<byteit101[m]>
oh good, i'll commit that
<headius>
I don't know what other odd platform names we'll come across
<headius>
dunno why this one in particular does it differently
<headius>
the versions make it a little tricky... ffi-binary-libfixposix requires 0.5.0 of bindings, but that is only a pre gem
<byteit101[m]>
though lfp is truly dev
<byteit101[m]>
yea, thats for dev only
<byteit101[m]>
but i was thinking 0.4.8 as a fake version
<byteit101[m]>
i havent actually "installed" them yet, just copied the files around or used -I
<byteit101[m]>
thats othersise on my todo list
<headius>
So where is that bizarre version number coming from
<headius>
Being able to build those binaries will be good for distributing in JRuby, but I'm not sure how necessary it is for typical gem usage if we just make the gym build the library on install
<byteit101[m]>
then versions should be less weird
<headius>
Like the sass gem just builds libsass on install
<headius>
Ok I will try that
<byteit101[m]>
(export it in your shell)
<byteit101[m]>
unsure why top rake not doing that
<headius>
what path should that point at?
<byteit101[m]>
libfixposix.so
<byteit101[m]>
the one that was built
<byteit101[m]>
require_relative './ffi-binary-libfixposix/lib/libfixposix/binary/version' has it
<headius>
didn't seem to fix the weird version, what was it supposed to do?
<headius>
I updated it to the same path but still same result
<byteit101[m]>
does it work if you rake build in ffi-bindings directly? not top level
<headius>
I can give that a try in a minute
<byteit101[m]>
k
<headius>
As far as reviewing stuff I don't know if there's much more we need to cover. Build issues notwithstanding, the breakdown and API all seem logical to me
<headius>
Is there a reason you don't want a gem that builds libfixposix on install?
<byteit101[m]>
not really, other than it's nice if we don't require a working c compiler
<headius>
I mean it's obviously nice to have pre-built stuff for JRuby but it isn't strictly necessary if you are running a typical Ruby environment
<headius>
Yeah okay
<byteit101[m]>
Another thing to review beyond building and CI building on arm64 apple, etc, is the native library loading
<headius>
So the question of how to distribute the binaries with JRuby jars comes back to that open question we don't have an answer for just yet
<byteit101[m]>
complete jar is easy
<byteit101[m]>
because I got the native library loading thing
<headius>
I think we can punt on that for the moment if we have this gem available that people can install to upgrade their spawn logic
<byteit101[m]>
I couldn't figure out how gems may be loaded from a jar though
<byteit101[m]>
*non-complete jar
<headius>
Any version of the jar will honor the gem environment variables for path and so on
<headius>
They will use the built-in gems by default unless you override that with one of the envs
<headius>
The problem is you can't load a native binary from within the gem
<byteit101[m]>
When?
<headius>
JFFI has some tangled and not great logic for unpacking the binary to a temp location as you have seen
<byteit101[m]>
(rather, when are gems not unpacked?)
<headius>
In the case of the complete jar, the gems are never unpacked
<byteit101[m]>
Oh I solved that with my native library loading branch
<headius>
Ok branch on this same repo?
<byteit101[m]>
my jruby PR is on top of the branch that solves that in case you want to merge them separately, holdon
<headius>
What you have is fine, but it duplicates logic within JFFI itself (or possibly that is in jnr-ffi) and there are security implications to throwing an executable into an arbitrary temp directory
<headius>
It's just been a to-do for a long time to settle on one standard way of unpacking and managing these native binaries
<byteit101[m]>
Hmm... So should I replace mine with that? or leave it in for now and clean it up later?
<headius>
I think we stall on that answer and evaluate the available options
<headius>
At the very least pulling out a jnr-native-library API for selecting and loading archived binaries, merging the existing JFFI code with what you have here
<headius>
So then we could just use the same API that JFFI uses to ship and unpack other binaries
<byteit101[m]>
Sounds good
<byteit101[m]>
How should I go about building in CI/for the jar? I know someone was looking at dockcross, and I still would like to have a build of apple arm64