01:53 < sreenik> Hi, I am trying to use mlpack with nvblas. Simply linking with -lnvblas is not showing any cuda profiling. (How) should I change armadillo's config.hpp? Should I add a #define ARMA_USE_NVBLAS?

02:01 < sreenik> Sorry that was a useless idea. Tried it to no avail.

02:09 < sreenik> [This] https://github.com/mlpack/mlpack/issues/1610 probably discusses some approaches but I am not sure where I am going wrong.

04:45 < Xain> hey my name is zain from Manipal and I'm interested in the GSoC project on NEAT. I've read up on NEAT and it's variants, but I haven't see anything on using it as an optimizer. Could you show me where it has been used that way?

04:47 < xain> zoq: I think you might have missed my message, so I'm reposting here.

06:14 < niteya> Hello , For gsoc I was thinking of implementing automatic differentiation for differentiable and differentiable separable functions using jets so please let me know if this is a good idea. It seems my previous comment got mixed in between a conversation.

13:17 < rcurtin> niteya: AD is very popular and hot right now, but if you take a look at the ensmallen paper, it often underperforms compared to a hand-implemented gradient method

13:18 < rcurtin> so I think that for a library like ensmallen which is focused on the speed of the implementation, it's reasonable to expect people to implement their own objective and gradient

13:18 < rcurtin> and, if they don't want to, I think they could use one of the existing C++ toolkits for AD to do that

13:18 < rcurtin> I do think it might be nice to add a little tutorial or documentation somewhere about using an existing C++ AD library with ensmallen... but that's probably not be enough for a GSoC project :(

14:21 < Suryo> zoq: I have an important update for you

14:21 < Suryo> I've implemented a method in PSO that can handle constraints

14:23 < Suryo> Following rcurtin's recommendation to preserve the current API style for constrained optimization provlems, which is documented here: https://ensmallen.org/docs.html#constrained-functions

14:24 < Suryo> I got the idea that one way of handling constraints in PSO would be to incorporate the constraints into the objective

14:26 < Suryo> I mean, of course I'm aware of how constrained Optimization is done analytically, but I kept imagining that we should define function classes for each constraint and repeatedly check if the particles satisfy them.

14:28 < Suryo> One way of handling constraints in PSO is to preserve the feasibility of the particles over iterations. That's the approach that I had investigated in the past.

14:28 < Suryo> In this, your initial set of particles need to be feasible.

14:29 < Suryo> So what I've done is basically wrote another initialization class, and while initializing particles, i evaluate each of them to make sure they're feasible.

14:29 < Suryo> If a particle is not feasible, it is evaluated as infinity. That's a standard approach.

14:30 < Suryo> So when particle positions are updated, the method will basically not update a particle if it's position is unfeasible

14:32 < Suryo> Now unlike what I had discussed earlier, i didn't template the feasibility checks separately. In a way, the feasible updating is naturally taken care of if infeasible positions have a value of infinity

14:32 < Suryo> I've also written two tests and updated my branch and you can see in PR#86 that the tests are passing.

14:32 < Suryo> At this point, I'd request you to review my PR. Also, let me know what you think about this

14:33 < Suryo> Having implemented this, I'm left with many more questions than i began with. I will discuss them with you soon.

14:33 < Suryo> I'm going to be unavailable for most of today. But i will intermittently check my messages.

14:34 < Suryo> Last thing - i ran into problems with the SPSA tests on my computer again.

14:34 < Suryo> Sometimes, the SPSA tests pass, sometimes, they don't.

14:34 < Suryo> Thanks.

14:48 < favre49> I ran the SPSA logistic regression test with random seeds and found that the test fails pretty often with accuracies below 60

14:55 < favre49> Also, I've read a lot more about topics related to the NEAT project but i still cannot find anything that could be used as an optimizer. Could you please point me in the direction of some papers that do so? Thanks in advance

15:14 < rcurtin_> Suryo: favre49: really, the SPSA test is still fragile? I thought we'd fixed that... let me try to reproduce shortly

15:17 < favre49> I read your conversation on the commit, and saw you used arma::arma_rng::set_seed(std::time(NULL)). I used arma::arma_rng::set_seed_random().

15:17 < favre49> I don't see how that would make a difference unless you ran it multiple times in the same second while testing, but i thought i should tell you anyway

15:40 < rcurtin_> yeah, should be the same thing

16:54 < Suryo> rcurtin: yeah, it kind of is fragile. The PSO tests passed in my computer, the spsa failed. I updated my PSO dev branch and just had my fingers crossed hoping that travis would pass all the tests haha

16:55 < rcurtin_> we can merge anyway even if the SPSA test fails, since it's clearly not any of your code causing the problem :)

16:55 < rcurtin_> it'll be this afternoon until I can try to reproduce and fix it though

16:55 < rcurtin_> today is meetings from me basically straight from 9am to 3pm... I'm not sure how my life got so full of meetings but I can say I'm not a huge fan...

16:59 < Suryo> Yes of course. But the green ticks are satisfying to look at :)

16:59 < Suryo> Just kidding

17:02 < rcurtin_> agreed :)

17:44 < dtrix> hi

17:45 < dak_jay_> hi

20:40 < yogesh> hey guys!

20:54 < sreenik> rcurtin: I was playing around with the DigitRecognizer.cpp under the models repo. Applying those changes and replacing the sigmoid activations with relu, I reached an accuracy ~97-98% on the validation set. However, the model just wouldn't converge if I normalised the training set. On analysing, I realized that most of the values of the arma::normalised matrix were set to 0. Any idea on why this is happening?

20:55 < rcurtin> is it being divided by an integer somewhere maybe?

20:55 < rcurtin> yogesh: hi there!

20:56 < rcurtin> Suryo: favre49: yeah, I see the failures for the SPSA test now... actually I think when I ran it, I grepped the output for 'fail' but it should have been "FAIL", so I never saw the failures :)

20:56 < sreenik> rcurtin: May be so, I am using arma::normalise

20:57 < sreenik> Another thing, whenever I run the CNN version of it, the CPU usage is limited to just one core, whereas all 4 cores are used in the case of the simpler DigitRecognizer.cpp

21:01 < sreenik> I think I should test it with some more models. That should give a better picture

21:05 < rcurtin> sreenik: hmm, are you using OpenBLAS? that should use multiple cores

21:05 < rcurtin> but it depends on where exactly the bottleneck is

21:06 < rcurtin> I think when I did the normalization I just divided the matrix by 255.0, since all values are between 0 and 255

21:06 < rcurtin> not sure what arma::normalise() does exactly

21:07 < sreenik> I have done that too, divide by 255.0. Same result

21:07 < sreenik> Yes, linked with nvblas and openblas

21:09 < rcurtin> oh, wait, sorry, I should have thought a little harder

21:09 < rcurtin> the MNIST images are mostly black, so they would mostly have values of 0

21:09 < rcurtin> for convergence, you might try playing with the step size of the optimizer

21:10 < sreenik> Well that's right. Might see that one

21:18 < zoq> xain: Sorry for the slow response. The idea loosely follows the idea behind the "Learning to learn by gradient descent by gradient descent" paper, "Is the meta-EA a viable optimization method?" and "Comparing Evolutionary Algorithms for Deep Neural Networks" is another interesting one. Rahul also pointed me to the "NeuroEvolutionary Meta Optimization" paper.

21:40 < SinghKislay> Hi