07:37 < partobs-mdp> How can I extract gradient from SGD optimizer history? (complete list of gradients or maybe extract it on every iteration)

07:38 < partobs-mdp> (I need it for unit-testing gradient clipping)

08:09 < partobs-mdp> partobs-mdp: By the way, I have benchmarked clamp() vs transform(). More here: https://github.com/mlpack/mlpack/pull/1070#discussion_r128972379

08:10 < partobs-mdp> (Why would I ever want to send a message to MYSELF in a public chat room oO)

09:35 < zoq> partobs-mdp: Unfortunately, there is no option to get the gradient from the optimizer, but I think you could just test the clipping policy by passing a generated gradient.

11:33 < lozhnikov> kris1: I just looked through your blog post. Actually, the 8th week is over. So, I suggest to replace "Week 7" by "Week 8".

11:42 < kris1> Okay i will change that accordingly.

12:30 < partobs-mdp> zoq: I thought a little bit and found a way to write a proper unit test for clipping. True, we can't sneak peek our gradient in SGD, but we're not force to use SGD interface - we can just call Update methods directly and see if they correctly update parameter values.

12:30 < partobs-mdp> zoq: For vanilla updates it works

12:31 < partobs-mdp> Compiling & testing for momentun updates

12:32 < partobs-mdp> About the CrossEntropy test: should it be a test with a simple neural network just to check that it works together, or should it be a test checking the outputs for given inputs (e.g., it should check that [.5, .5, .5, .5] input yields 4ln(2) error value)

12:32 < partobs-mdp> What do you think?

12:35 < zoq> partobs-mdp: Right, the optimizer class is basically just a wrapper, for the parameter, gradient and some additional information about the optimization process.

12:35 < zoq> partobs-mdp: For me testing it on some generated data is fine e.g. as you said test it on [.5, .5, .5, .5], but if I like you could also add another test that uses some simple network.

12:37 < partobs-mdp> zoq: And where can I find a test for MeanSquareError? I couldn't find anything like that in ann_layer_test.hpp.

12:38 < zoq> partobs-mdp: I think there is no test, I should fix that.

12:41 < partobs-mdp> zoq: When you write a test for MeanSquaredError, could you show it to me? I just can't completely figure out how to write a test for CrossEntropy (but MeanSquaredError test would really help since they're both loss functions)

12:43 < zoq> partobs-mdp: Okay, I can write a test later today, should be straightforward to adapt the test for the CE layer.

17:09 < sntgpt> hi

17:17 < sntgpt> I would like to contribute in manner. I am not sure how to start.

17:17 < sntgpt> *in some manner

17:23 < zoq> sntgpt: Hello, have you seen http://www.mlpack.org/involved.html or http://mlpack.org/gsoc.html, and on that note we are always open for new contributions.

17:28 < sntgpt> Thanks, I'll have a look

17:31 < ironstark> zoq: I was having a look at dlib-ml's documentation

17:32 < zoq> ironstark: Sounds good, what do you think?

17:33 < ironstark> They only have svm and kmeans implemented , is that correct?

17:34 < zoq> I think I have seen neighbor search and logistic regression as well.

17:35 < zoq> If we go for dlib-ml we should make some sort of list.

17:44 < ironstark> http://dlib.net/ml.html Here I can see only the svm's and some clustering methods

17:45 < zoq> ironstark: Also, I think we don't have to implement "everything" at least not now.

17:45 < zoq> rcurtin: Do you have any method preferences?

17:45 < zoq> ironstark: I'll have to step out, I'll take a look at the link once I get back.

17:47 < ironstark> zoq: Sure. In the meantime I will work on understanding the library. Also should we implement using C++ or Python?

17:47 < ironstark> as both are available

18:34 < zoq> ironstark: So looks like there is a function for approximate nearest neighbors search and nearest neighbors search: http://dlib.net/term_index.html

18:35 < rcurtin> zoq: ironstark: sorry for the slow responses, I was unavailable from Friday through Sunday and I am just now catching up

18:36 < rcurtin> I think benchmarking R is important but I don't think the order is important so if you do dlib-ml before R that's fine with me

18:36 < rcurtin> give me a little while and I will have an in-depth response for what methods I think are good to benchmark from dlib-ml

18:38 < zoq> rcurtin: Sounds good, I guess kmeans, neighbor search, "svm" are methods that I think would be interesting.

18:39 < zoq> About python or c++, I would go with c++, but if you think python is easier go with python.

18:39 < ironstark> rcurtin: zoq: So I will start working on these three implementations first.

18:40 < ironstark> About python or c++, I will look up the documentation and implementation examples and will implement accordingly.

19:08 < rcurtin> ironstark: there is a deep neural net toolkit too, so maybe the MLP implementation could be useful also

19:09 < ironstark> rcurtin: okay, I will look into it.

19:11 < rcurtin> the other three ideas would be good, but I would use both dlib's exact NN search, the approx one, and the LSH one (it has three implementations)

19:13 < ironstark> okay

22:19 < zoq> pretorium[m]: Here is the simple test: https://github.com/mlpack/mlpack/pull/1071