ChanServ changed the topic of #mlpack to: "mlpack: a fast, flexible machine learning library :: We don't always respond instantly, but we will respond; please be patient :: Logs at http://www.mlpack.org/irc/
< rcurtin[m]> Andrew Spott (Gitter): https://github.com/mlpack/mlpack/pull/2756 -- it'll be a while until that's in a release in `Pkg`, but you could compile the Julia bindings locally with that patch, and it should solve the issue
< AakashkaushikGit> > `zoq on Freenode` I hope this makes sense, let me know if there is anything that I should clarify or if that does not build.
< AakashkaushikGit> Hi @zoq, Thank you so much for taking out the time to look at this, I will take a look and let you know.
ImQ009 has joined #mlpack
< zoq> AakashkaushikGit: Sounds good, I bascially replaced the visitor use inside the init network class.
< AakashkaushikGit> Hey, even i tried the same but got stuck with the last visitor. I should be able to run this fairly quickly now. Thanks to you :D
< AndrewSpottGitte> @ryan: Thanks! Do you know where I can find documentation to compile the Julia Bindings?
< rcurtin[m]> Hmm, you should be able to just follow the usual build instructions in the README and specify the CMake flag `-DBUILD_JULIA_BINDINGS=ON`; then, after `make` completes, you'll have a directory `build/src/mlpack/bindings/julia/mlpack/` that contains the package (and compiled libraries)
< RishabhGarg108Gi> I finally overcame the hurdles I was facing with cereal serialization for the parser. As we know that cereal can read and write JSON, so I was wondering if there was some parser within cereal for doing that. And I was actually right. Cereal uses RapidJSON library for handling JSON. So, I simply made use of that and I have done around 50% of the work on it.
< RishabhGarg108Gi> I also wrote a very minimlistic version of boost::variant for storing the parameters.
< rcurtin[m]> I'm not sure what the best way to install that is. You could overwrite everything in `.julia/packages/.../mlpack/`... or you could just try running `julia` with `JULIA_PROJECT` set to that build package directory. There are a handful of options
< rcurtin[m]> The way it actually gets built and distributed is a little bit complex; the compiled libraries come through the `mlpack_jll` package, built on Yggdrasil, and then the `mlpack` package just contains the `.jl` files; but if you build by source, you get both of those at once, so it's a bit different
< RishabhGarg108Gi> Now everything seems simple, but I am facing a small issue. For every parameter, to access its data i need to pass its type name for example, if it the parameter is of type int, then I have to write something like `param.get<int>()` to get its value. But the type of each parameter may not be known to us each time directly. So I want to call something like `param.get()` and I can get its stored data. So, can someone
< RishabhGarg108Gi> please help me how to do that. I am attaching the code file [here](https://drive.google.com/file/d/1iaNSU1xQVnwtE3KSQkrJ_9QecaY5Itlm/view?usp=sharing). Maybe whatever I just said make sense once you look at this file. Thanks.
_slack_mlpack_US has joined #mlpack
_slack_mlpack_US is now known as GauravGhati[m]
< GauravGhati[m]> Hey everyone, I'm new in organization, and trying to run a few sample codes, but I'm getting error while running executable of sample codes. `./a.out: symbol lookup error: ./a.out: undefined symbol: arma_H5T_NATIVE_DOUBLE` I'm unable to fix it, could anyone help?
< AndrewSpottGitte> @ryan: so I built MLPack from source with your changes, replaced everything in `.julia/packages/mlpack/.../src/` with the stuff in `build/src/mlpack/bindings/julia/mlpack/src`, and the stuff in `.julia/artifacts/.../include/mlpack` with the stuff in `build/include/mlpack`, and `.julia/artifacts/.../lib/libmlpack.so*` with the stuff in `build/lib/libmlpack.so*`. Unfortunately, on running my code I'm now getting a
< AndrewSpottGitte> fatal error: `[FATAL] Parameter --reference (-r) is defined multiple times with the same identifiers.`
< AndrewSpottGitte> any idea how to debug this?
< rcurtin[m]> intersting... I think you installed everything right, can you provide a backtrace or anything?
< rcurtin[m]> *interesting :)
< rcurtin[m]> what were you trying to run in Julia when this happened?
< AndrewSpottGitte> where should I post the backtrace?
< rcurtin[m]> you could post it here, or pastebin, wherever is convenient
< rcurtin[m]> I see, and it segfaults and crashes Julia? Sorry about that, I know that is frustrating
< AndrewSpottGitte> yea
< rcurtin[m]> let me try reproducing this later today; did you make any other changes to the mlpack source code before you ran?
< AndrewSpottGitte> I did not
< rcurtin[m]> 👍️
< AndrewSpottGitte> Thanks! I'm expecting it is part of the installation (I didn't remove the mlpack_jll package... but it isn't relevant anymore?)
< rcurtin[m]> yeah, I think it is okay to leave the mlpack_jll package there (so long as you didn't have a Julia session still running from when you had the old version of the mlpack package)
< AndrewSpottGitte> I didn't.
< AndrewSpottGitte> (I don't think...)
< rcurtin[m]> well if it segfaults when mlpack throws an exception then probably not :-D
< AndrewSpottGitte> actually, it looks like I did...
< AndrewSpottGitte> I'm re-running it after I closed the other julia session.
< rcurtin[m]> oh my... hm. so, let me play with the KDE binding later today and see if I can get it to crash
< AndrewSpottGitte> thanks
< AndrewSpottGitte> let me know if I can help at all.
< rcurtin[m]> it's possible that this might have to do with the installation not being set up quite right (but I'm not enough of a julia expert to know for sure)
< rcurtin[m]> maybe can you try this super simple example and see what happens?
< rcurtin[m]> (just in a clean Julia session)
< AndrewSpottGitte> using mlpack; x = rand(10, 10); y = rand(10); output = mlpack_lars(input=x, responses=y, verbose=true)
< AndrewSpottGitte> ERROR: UndefVarError: mlpack_lars not defined
< AndrewSpottGitte> (edited) using mlpack; x = rand(10, 10); y = rand(10); output = mlpack_lars(input=x, responses=y, verbose=true)
< AndrewSpottGitte> ERROR: UndefVarError: mlpack_lars not defined => `using mlpack; x = rand(10, 10); y = rand(10); output = mlpack_lars(input=x, responses=y, verbose=true)`
< AndrewSpottGitte> `ERROR: UndefVarError: mlpack_lars not defined`
< rcurtin[m]> oh oops, use `mlpack.lars` instead, sorry!
< AndrewSpottGitte> (mlpack.LARS(Ptr{Nothing} @0x0000000000d529c0), 0×0 LinearAlgebra.Adjoint{Float64,Array{Float64,2}})
< rcurtin[m]> ok, awesome, that worked---so maybe the issue is specific to KDE
< rcurtin[m]> anyway, I have some other stuff I have to handle this afternoon, but I'll let you know what I find out when I'm able to look into it 👍️
< AndrewSpottGitte> `mlpack.kde(query=rand(10,10), reference=rand(10,10))`
< AndrewSpottGitte> `(mlpack.KDEModel(Ptr{Nothing} @0x00000000015019e0), [3.5218816829026456e-5, 4.3311416507770994e-5, 4.3648697206800735e-5, 4.297213284714002e-5, 5.447050870072033e-5, 4.587582433175843e-5, 3.289868430401761e-5, 4.987510236058251e-5, 4.231670700398405e-5, 3.590605327744949e-5])`
< AndrewSpottGitte> I'll keep looking into it, maybe I can narrow the problem down.
< AndrewSpottGitte> If you have ever run into the "[FATAL] Parameter --reference (-r) is defined multiple times with the same identifiers." error, maybe I can backtrack that to see what I'm doing wrong.
< rcurtin[m]> yeah, so that actually would be an issue on the mlpack side, where somehow some parameter got defined twice; but our CI tests should pick up problems like that...
< AndrewSpottGitte> weird.
< AndrewSpottGitte> I can't get that error to pop up again (they are now all the short version without the stack trace).
< rcurtin[m]> when you say the "short version", do you mean the `(mlpack.KDEModel ...` output?
< AndrewSpottGitte> no, the `signal (11): Segmentation fault ...` message.
< AndrewSpottGitte> I'm getting a segfault, and it is showing where it happens, but I'm not getting the `[Fatal]` message, or the full stack trace.
< rcurtin[m]> ah, okay 👍️
ImQ009 has quit [Read error: Connection reset by peer]
< AndrewSpottGitte> @ryan: Some more information. I turned on `verbose` in the `kde`function, and I get the same code to run multiple times before it fails. I then started printing the inputs to the function before calling the function, and waiting for it to fail. I took the inputs that I saved and reran kde with those inputs... and it doesn't fail. my current guess is that the crash is happening somewhere in the finalizer... and
< AndrewSpottGitte> it is non-deterministic.
< AndrewSpottGitte> The kde function was called a whole bunch. In this particular case, it was called with an input_model.
< rcurtin[m]> nice digging! I think I see what might be going on
< rcurtin[m]> strange to see a stack overflow in the finalizer, but I bet that what's happening here is that when we call `kde()` with an input model (which is a Julia object that already has a finalizer), we create a new output model and create a finalizer for it too---even though it points to the same C++ memory, and so it gets freed twice...
< AndrewSpottGitte> ah~
< AndrewSpottGitte> (edited) ah~ => ah!
< AndrewSpottGitte> That makes sense.
< rcurtin[m]> I bet after you call `kde()` with an input model, if you call `finalizer(k -> nothing, k)` for `k` as the output model, there will be no issue
< AndrewSpottGitte> and it is happening at weird times because Julia is garbage collected... so the memory can get freed pretty much whenever.
< rcurtin[m]> (assuming that I can call `finalizer()` a second time on an object and it will overwrite any existing finalizer)
< rcurtin[m]> anyway, this should give me enough to make a reproducible test case that I can add to our test cases, I think
< rcurtin[m]> but it may take a little while to think about what the right solution here is...
< AndrewSpottGitte> Yea.
< zoq> Starship high altitude test second try in 4 minutes :)
< ryan[m]> yeah, I saw that just a moment ago, opened up the stream and I'm watching now... hopefully more exciting today :)
< zoq> hold doesn't mean abort, I guess
< ryan[m]> hopefully...
< rcurtin[m]> yep... well I'm really glad you pointed this out... I didn't even think about this possibility when I was implementing the finalizer
< AndrewSpottGitte> so, it looks like `finalizer` adds a new `finalizer` function rather than overwriting the old one...
< rcurtin[m]> crap... well, that makes my life a lot harder :(
< rcurtin[m]> thank you for checking into that!
< AndrewSpottGitte> yea, mine too.
< rcurtin[m]> I think that I can solve this basically by implementing a crappy version of reference counting on the C++ side...
< AndrewSpottGitte> that would work.
< rcurtin[m]> "crappy" in that it will not be very feature-complete, not in that it wouldn't work :-D it should work fine I think, just a matter of setting it up right
< rcurtin[m]> anyway, needs some thought... I'll let you know when I have a new idea implemented
< rcurtin[m]> if you want a really short-term hack, you can modify all the types in `src/types.jl` and comment out the `finalizer()` lines in the constructors, and then add the `finalizer()` call manually after you run KDE (only once for each model)
< AndrewSpottGitte> That will work!
< AndrewSpottGitte> Thanks!
< rcurtin[m]> you'd still need to work on the branch that you built by hand though, because the `Delete()` functions aren't available in the version of mlpack that's published in the `Pkg` ecosystem
< AndrewSpottGitte> Can I do the modification on the installed files?
< AndrewSpottGitte> (or do I have to build again?)
< rcurtin[m]> I think you should be able to do it on the installed files; at least from the mlpack side no rebuild should be needed 👍️
< AndrewSpottGitte> btw: this link is dead: https://www.mlpack.org/doc/mlpack-git/kde_detailed-documentation
< rcurtin[m]> Huh, where did you find that link?
< AndrewSpottGitte> https://www.mlpack.org/doc/mlpack-git/julia_documentation.html#kde the "detailed documentation" link in the first paragraph.
< rcurtin[m]> Oh, strange, it must be generating the Markdown anchor wrong...
< AndrewSpottGitte> ah, just a few lines down.
< rcurtin[m]> it seems to be broken for all bindings and all languages... I wonder what went wrong? anyway, another thing to look into, thank you for reporting it :)
< rcurtin[m]> oh, hmm, the code for the finalizer might need to be modified because it's now not in the scope of the module... I think this could work:
< rcurtin[m]> ```
< rcurtin[m]> finalizer(x -> mlpack._Internal.lars_internal.DeleteLARS(x.ptr), model)
< rcurtin[m]> ```
< rcurtin[m]> but you'll probably need an `import mlpack` or `using mlpack` somewhere else, or something like this
< rcurtin[m]> hmm, why wouldn't the `mlpack` module be available when the finalizer is run though? I don't understand why the code you pasted wouldn't work
< rcurtin[m]> also yeah, `DeleteKDE` not `DeleteLARS`, sorry about that :) my mind autocompleted it wrong...
< AndrewSpottGitte> I'm running a short run while testing, and these are popping up at the very end...
< AndrewSpottGitte> it is possible that it is being run after it has unloaded everything.
< AndrewSpottGitte> (I haven't done a longer run to see if it still has an issue)
< AndrewSpottGitte> I just started a longer test. I'll let you know if it crashes the process.
< AndrewSpottGitte> so, it is popping up after each smaller run... but the program is still running.
< AndrewSpottGitte> The memory usage seems to be significantly lower, and the program completes...
< AndrewSpottGitte> It doesn't seem that the error is a problem.
< rcurtin[m]> strange!
< zoq> well, the first part looked good
< rcurtin[m]> they did way better than I usually do in similar situations in Kerbal
< zoq> haha
< rcurtin[m]> it was super impressive how quickly it righted itself
< zoq> yes
< rcurtin[m]> and honestly the explosion was pretty cool looking too :)
< zoq> yeah, but a landing would be cool as well :)
< rcurtin[m]> :)
< rcurtin[m]> hopefully the next test flight is soon; that was a lot of fun to watch
< zoq> yes, was nice to watch