verne.freenode.net changed the topic of #mlpack to: http://www.mlpack.org/ -- We don't respond instantly... but we will respond. Give it a few minutes. Or hours. -- Channel logs: http://www.mlpack.org/irc/
< rcurtin>
zoq: thanks, hopefully that will fix some issues
< rcurtin>
I will start the monthly build just to see :)
< rcurtin>
I see it's only 35C in Tucson where masterblaster is, so hopefully we can help make it a little warmer
vivekp has quit [Ping timeout: 276 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 248 seconds]
vivekp has joined #mlpack
< rcurtin>
manish7294: unfortunately I wasn't able to look into the LMNN issue at all today; I hope to have some more time tomorrow. sorry about that
< rcurtin>
my only thought is, maybe it's worth seeing if you can find a different implementation of LMNN or some SDP with slack variables that need to be updated in order to give you some ideas
< rcurtin>
alternately, there is also the idea of implementing a "wrapper" matrix class that just holds an arma::mat for the M matrix and then an arma::vec for the slack variables
< rcurtin>
but that gives a problem for LRSDP, which is that the diagonal part of the matrix means that the overall matrix being solved is going to be high-rank, thus violating the idea of LRSDP (low-rank)
< rcurtin>
so it's probably better if we figure out a way to handle the slack variables separately
< rcurtin>
I will read more tomorrow
< zoq>
Forgot to install cppcheck ...
vivekp has quit [Ping timeout: 260 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 256 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 255 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 255 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 255 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 256 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 255 seconds]
vivekp has joined #mlpack
sumedhghaisas has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 265 seconds]
wenhao has joined #mlpack
sumedhghaisas has quit [Read error: Connection reset by peer]
sumedhghaisas has joined #mlpack
wiking has quit [Read error: Connection reset by peer]
wiking has joined #mlpack
< wiking>
rcurtin, around?
< rcurtin>
just woke up... give me 20-30 minutes and I'll be back
< wiking>
kk
< wiking>
nothing urgent :)
< rcurtin>
ok, back. what's up?
sourabhvarshney1 has joined #mlpack
manish7294 has joined #mlpack
< sourabhvarshney1>
zoq: I have started with the BRNN work. I was busy with some travelling and home stuff. Will open a PR by tomorrow : )
< manish7294>
rcurtin: In the meantime, while wiking is not here, shall I disturb you?
< rcurtin>
manish7294: I left some messages last night, did you take a look at them? I haven't done anything since then except sleep...
< manish7294>
rcurtin: Yup! I went over them
< manish7294>
rcurtin: According to my findings there is only a single way to add slack constraints which is block diagonalization (what we are currently doing)
< manish7294>
sadly, I didn't find any method to reduce the complexity
< manish7294>
As far other LMNN implementation, I think I have scanned majority of them over last months and they all implement the same algorithm which author proposed :(
< rcurtin>
the block diagonalization strategy is almost certainly incompatible with the fundamental assumptions of LRSDP, therefore, it seems we will need to develop some kind of iterative algorithm
< manish7294>
The slack part in itself is a LP problem
< manish7294>
I am wondering, can something like combining to optimizers be done. It may sound stupid though!
< manish7294>
*two
< rcurtin>
no, I don't think that sounds stupid, I think that is the approach we'll have to take
< rcurtin>
what I have in mind right now would be something where you (1) take an SDP step (2) update the slack variables (3) return to step 1 until convergence
< rcurtin>
or something to this effect
< rcurtin>
really the slack variables are used as a penalty because it's not possible to fully satisfy all the constraints
< zoq>
sourabhvarshney1: Sounds great, hope you had fun.
< manish7294>
That sounds cool :)
< manish7294>
rcurtin: But then SDP we will have to generate a extra constraint for SDP
< sourabhvarshney1>
zoq: Yup I loved the work. I am trying to do some more work before opening the PR : )
< manish7294>
rcurtin: That constraint should determine the dependece of two part.
< rcurtin>
manish7294: I'm not fully sure what you mean. in any case, I do think the optimization can be either expressed like I suggested above, or it may be possible to replace the slack variables entirely with a penalty to the objective
< rcurtin>
I will need to consider each idea carefully, but it will take some time before I am able to do that fully, so you should also spend some time looking into the issue---don't just wait on my input :)
< manish7294>
rcurtin: That's right! I myself was not clear while writing that message, but as for SDP what will be the constraints then
< manish7294>
rcurtin: Don't worry, I am continously looking for this from past 3 months :)
< rcurtin>
I need to think about it, but to me, the constraints seem a little odd. each of the slack variables must be set (for each i-j-l tuple) such that the constraint is satisfied
< rcurtin>
so, I wonder, can the slack variables be removed from the constraints and be replaced with something similar to the LHS of the slack variable constraint in the objective?
< rcurtin>
but like I said, it needs some amount of thought, and I need to review the existing LMNN MATLAB implementation to understand it better
< manish7294>
rcurtin: Originally, slacks variables weren' t there. They are just an alias for constraint generation.
sourabhvarshney1 has quit []
manish7294 has quit [Quit: Page closed]
sumedhghaisas has quit [Read error: Connection reset by peer]
sumedhghaisas has joined #mlpack
< rcurtin>
manish7294: I know what my strategy for approaching the problem is; maybe if I write that down it will be helpful to you
< rcurtin>
first, I want to derive what a single update step for the augmented matrix [M 0; 0 s] (where 's' is the diagonal matrix of slack variables) looks like
< rcurtin>
my intuition is that the updates to each element of 's' will basically be whatever update is needed to satisfy the constraint (or some otherwise simple gradient step)
< rcurtin>
or, more specifically, that the gradient for each element of 's' will turn out to be something that is basically the same as just the gradient of that penalty term for the objective
< rcurtin>
if that is true, then the slack constraints are quite literally just a way to stuff the "margin" idea into an SDP, and if the penalty were just placed in the objective alone and optimized that way, there would be no actual need for the slack constraints
manish7294 has joined #mlpack
< rcurtin>
if the slack constraints can indeed be seen as superfluous, then the only remaining constraint is the positive semidefiniteness of the matrix
< rcurtin>
and if we then solve for L where M = L^T L (so, L may be low rank, but it is not required to be positive semidefinite), then we actually have an unconstrained optimization that is not an SDP!
< rcurtin>
that's a big long chain of dependencies to investigate, but if those things are all true (or, if something similar enough to LMNN can be expressed in that way), we can use mlpack's off-the-shelf solvers like L_BFGS and SGD variants
< rcurtin>
anyway, I hope the thoughts are helpful. I haven't dug into the reality of any of those conjectures yet though
< manish7294>
rcurtin: I think all of that make good sense.
< rcurtin>
the starting place would be deriving what a single-step update to the augmented matrix [M 0; 0 s] is
< manish7294>
rcurtin: As for s gradient step, the original LMNN implementation re-calculates impostors after every step.
< manish7294>
It thought it would be helpful to mention that.
< manish7294>
And one more important point is that target neighbors remain fixed. So only impostors are recalculated
< rcurtin>
this makes sense. it may be possible to add some optimizations to reduce the runtime needed for that kNN step to calculate the new impostors
< rcurtin>
but I think that is something we can deal with later :)
< manish7294>
Ya, that can be added at back of priority list, hahaha :)
< rcurtin>
also, in the absolute worst case, where we can't think of anything clever to do here, I think the right thing to do is implement a fast C++ version of the LMNN solver and then use this to implement BoostMetric (without LRSDP or other optimizations)
< rcurtin>
but... I spent a little while thinking about it, and I think it is possible to get some really nice speedup by re-expressing the LMNN optimization to remove the slack variables or some other trick
manish7294 has quit [Ping timeout: 260 seconds]
manish7294_ has joined #mlpack
< manish7294_>
rcurtin: I think we won't be able to use LMNN solver to solve BoostMetric , as they have redfinied the LMNN gradient step to exponential one
manish7294_ has quit [Client Quit]
< rcurtin>
manish7294_: fair, sorry for the error on my part
manish7294 has joined #mlpack
< manish7294>
rcurtin: But if we can bring speed ups to LMNN solver, I think that would be good enough.
manish7294_ has joined #mlpack
manish7294 has quit [Ping timeout: 260 seconds]
< rcurtin>
manish7294: agreed
manish7294_ has quit [Ping timeout: 260 seconds]
ImQ009 has joined #mlpack
manish7294 has joined #mlpack
< wiking>
rcurtin, oh sorry haven't seen your ping
< wiking>
i was wondering if you know some details about boost serialization fw regarding templates
< rcurtin>
I know boost serialization fairly well, how can I help?
< wiking>
do you need to do type registration for supporting templated classes?
< wiking>
say i have something like template<typename T> class A {... }; where T is not a primitive type
< wiking>
and i wanna be able to serialize A
< wiking>
as well as deserialize
< rcurtin>
when we do this, we just make sure that T is serializable and there are no problems there
< wiking>
mmm
manish7294_ has joined #mlpack
< manish7294_>
zoq: rcurtin: There is still python-cython written instead of cython in mlpack 3.0.0 debian dependencies. I thought it would be good to point it out.
manish7294 has quit [Read error: No route to host]
vivekp has quit [Ping timeout: 255 seconds]
manish7294 has joined #mlpack
manish7294_ has quit [Ping timeout: 260 seconds]
manish7294 has quit [Ping timeout: 255 seconds]
manish7294 has joined #mlpack
< rcurtin>
manish7294: I spent some more time thinking about it and reviewing the details of your proposal, and I want to be careful not to force you to change your idea too much if you don't want to
< rcurtin>
so, while I think that it's possible to reformulate the LMNN optimization (and similarly the BoostMetric optimization) to remove the need for the "augmented" matrix [L 0; 0 \epsilon] (in the terminology of your proposal)
< rcurtin>
I also think it's possible that the idea given in your proposal could also work, but the efficiency of the implementation would hinge on making some "wrapper class" that can efficiently hold [L 0; 0 \epsilon]
< rcurtin>
which do you think would be better to pursue? if you'd prefer to stick with your approach, I won't sit down and try to do derivations on the LMNN objective function :)
< manish7294>
rcurtin: Hosnestly, I had my doubts related to LMNN SDP approach scalability from the starting.
< manish7294>
rcurtin: I would prefer a approach that can do the job in best way.
< manish7294>
rcurtin: And is best for the end user.
< rcurtin>
manish7294: I think, for the end user, the API will look the same so that's no issue
< manish7294>
rcurtin: If we could even improvise existing algorithm to gain speedup that would be better too from my point of view.
< manish7294>
rcturin: you got the point but I reffered to benchmarks :)
< rcurtin>
true :)
< manish7294>
and memory too
< rcurtin>
for the scalability, I think it could be okay with your approach, but it will definitely need a custom matrix class so that the operation trace(C*R*R^T) is efficient
vivekp has joined #mlpack
< rcurtin>
(I think that's not the exact operation... but it's something like that, I don't remember off the top of my head right now)
< rcurtin>
the custom matrix class would be... semi-simple, but a decent amount of code
< manish7294>
ya but storing a large number of matrices in SparseA and with all of them having a huge size will definitely lead to problems at some point of time
< manish7294>
as a result we can see primal dual giving bad_alloc() on small dataset too
< manish7294>
while the same works for LRSDP
vivekp has quit [Ping timeout: 255 seconds]
< manish7294>
rcurtin: So shall we think more about two optimizers approach?
vivekp has joined #mlpack
travis-ci has joined #mlpack
< travis-ci>
mlpack/mlpack#4908 (master - a30f5f8 : Ryan Curtin): The build has errored.
< rcurtin>
KevinAvignon: hi there, I would say it is going pretty well so far :)
< rcurtin>
the channel is pretty active these days
< zoq>
Agreed, a bunch of really neat improvements, additions and discussions so far.
vivekp has quit [Ping timeout: 255 seconds]
< KevinAvignon>
@rcurtin: I was wondering, while I'm mentoring Yasmine, when I make progress on the C# automatic bindings, should I include progress reports on the development blog?
< KevinAvignon>
rcurtin: I was wondering, while I'm mentoring Yasmine, when I make progress on the C# automatic bindings, should I include progress reports on the development blog?
< rcurtin>
if you'd like to make blog posts too, definitely feel free! there is no restriction on who can write to the blog
< rcurtin>
although, actually, I need to get your github account so I can get your permissions set right for that repo
< rcurtin>
username Kavignon?
< KevinAvignon>
Yes my handle is indeed Kavignon! That's good! I wanted readers to have the perspective on the automatic binding from a student and from a mentor and see it from two different tech stacks to have a sound idea of how big a project it can be.
< rcurtin>
yeah, that would be great---let me get the permissions set up now
vivekp has joined #mlpack
< rcurtin>
ok, set it up, but I think you'll need to click a link in an email to confirm :)
< KevinAvignon>
I've accepted your invitation! As for the repo's permission, all you'd need would be to be a collaborator of my project or do you need something specfic?
< rcurtin>
yeah, you should be able to push to the blog repo now; I don't think that I would need to be a collaborator of a project of yours (maybe I misunderstood the question?)
< KevinAvignon>
I've misunderstood what you said. When you first mentioned permissions, I thought you were speaking of my binding repo* It's my fault
vivekp has quit [Ping timeout: 264 seconds]
< rcurtin>
ah, nah, sorry, I just meant permissions for the mlpack/blog repo, in case you wanted to push your blog writings to there so it gets displayed on mlpack.org/blog/
< KevinAvignon>
All right! Oh so you'll know, I think, for at least now, I'll meet with Yasmine for at least 2h a week. We'll work side by side so if she has any questions, I can answer them quickly.
< rcurtin>
sounds great, it is really nice that you are in the same place :)
< rcurtin>
I will be in Montreal in December for NIPS, so if you like, we can meet up
< rcurtin>
(maybe some other mlpack-ers will be there also?)
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 260 seconds]
vpal has joined #mlpack
KevinAvignon has quit [Quit: Page closed]
vpal has quit [Ping timeout: 240 seconds]
vivekp has joined #mlpack
vivekp has quit [Ping timeout: 240 seconds]
vivekp has joined #mlpack
haritha1313 has joined #mlpack
< haritha1313>
zoq: The complete structure of CF class is now done after refactoring based on decompositionpolicy. I was intending to make a PR today, but unfortunately I'm stuck up with some errors in the new policy classes when building..
< haritha1313>
So I guess I can start NCF only tomorrow.
< zoq>
Okay, no worries, I'll take a look at the PR once it#s open; I can also help with the build issue, if you need any help with that.
vivekp has quit [Ping timeout: 268 seconds]
vivekp has joined #mlpack
< haritha1313>
zoq: It looks solvable to me as of now - some header inclusion troubles, just that the error log is too long to comprehend easily :D . I'll let you know if there's any trouble, thanks :).
< zoq>
haritha1313: Right, using template doesn't beautify the output :)
vivekp has quit [Ping timeout: 240 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 256 seconds]
sumedhghaisas2 has quit [Read error: Connection reset by peer]
vivekp has joined #mlpack
sumedhghaisas has joined #mlpack
vivekp has quit [Ping timeout: 256 seconds]
haritha1313 has quit [Ping timeout: 260 seconds]
vivekp has joined #mlpack
< rcurtin>
manish7294: I spent a few minutes talking with a colleague about the LMNN reformulation today. we ended up writing on a whiteboard, and I think that the proposition in this image I took is correct: