14:32 < Atharva> rcurtin: zoq: what irc client do you use to stay connected forever? Do you use it on phone or pc?

14:35 < rcurtin> Atharva: I use irssi in a GNU screen session on a server that I host (it's the same server that hosts mlpack.org)

14:35 < rcurtin> when I read messages, I simply connect from whatever computer I am using with ssh and resume the screen session

14:36 < rcurtin> I think it is an unusual setup but it works for me :)

14:41 < Atharva> Oh, okay, I don’t have a server for that kind of a setup. I think I would have to just search the net for a good client.

14:42 < rcurtin> I know there are IRC bouncers out there... I want to say one of these is called 'matrix'? but I am not certain

14:53 < Atharva> Yeah, I have heard about the bouncers, not quite sure what they do. I will check them out. There are cloud based clients which keep you online all the time but they charge on a per month basis and are quite expensive.

14:53 < rcurtin> yeah, I thought there was at least one that was free

14:54 < rcurtin> but even if you don't have a way to have a client always in the room, it is logged to http://www.mlpack.org/irc/, so you can always keep an eye on that :)

14:56 < Atharva> That is extremely useful, I always use the logs.

14:57 < Atharva> I had this another doubt, how do I build everything else except the test framework. I am trying some changes in the mlpack ann codebase and some tests fail to compile because of that.

14:57 < rcurtin> you can configure cmake with -DBUILD_TESTS=OFF

14:58 < rcurtin> and then when you type 'make', by default, the tests will not be built (you should still be able to type 'make mlpack_test' if you want the tests)

15:01 < Atharva> It’s still failing, I think I will have to check the changes I made. How do I compile just the ANN module?

15:01 < rcurtin> well, so this one is a little bit tricky

15:01 < rcurtin> the ANN code doesn't actually compile into anything because it is header-only

15:01 < rcurtin> this is the case with a lot of code in mlpack (but not all of it---any .cpp files in src/mlpack/core and src/mlpack/methods get compiled into libmlpack.so)

15:02 < rcurtin> so the only way it gets compiled into something is either in the tests in src/mlpack/tests/ or in the bindings found in src/mlpack/methods/*/*_main.cpp

15:18 < rcurtin> ok... I think that I have mlpack pypi packages compiling successfully. once I verify they are working right I'll upload the scripts into the jenkins-conf repository

17:24 < s1998_> zoq: rcurtin: w.r.t PR 9 of models, I have changed the dataset to mnist (currently in csv but the train has size of 104MB, limit : 100MB). Should I read the data from original MNIST dataset (which was in bytes format) ?

17:25 < s1998_> Or should I break the train data (in csv) into two parts and then push the changes ?

17:27 < s1998_> Another thing is the current implementation reaches test accuracy of 82% but I think this can be fixed using (batch) normalization (since currently only l2 normalization is used). Should I do this (as in write code to find mean and sigma) or use batch norm layer ?

19:22 < zoq> Atharva: I use almost the same setup irssi + tmux.

19:22 < zoq> s1998_: Testing the byte format sounds reasonable to me, hdf5 might be another solution.

19:23 < zoq> s1998_: About the accuracy, I think there are multiple options we could test, different architecture, l2 norm, batchnorm, another optimizer, so if you like please feel free to look into some options.

20:32 < manthan> rcurtin : when exactly is the gradient() called for a differentiable layer?

20:33 < zoq> manthan: After the backward step.

20:33 < manthan> i mean what exactly will be the difference in the backward and gradient for a layer?

20:33 < manthan> backward will contain the update rule for backward pass

20:33 < manthan> what will gradient contain exactly?

20:36 < zoq> The update step for the parameter, you could merge both steps into one, but in this case you would have to run the backward step (error calculation) for the first layer as well, which is unnecessary since the error isn't going to be used.

20:38 < manthan> so the error obtained in this function is the error upto the present layer and we have to write the logic for updating the parameter given the error and input?

20:38 < zoq> correct

20:39 < rcurtin> zoq: would it be right to say that Backward() is the derivative of the inputs with respect to the error, whereas Gradient() is the derivative of the parameters with respect to the error?

20:39 < rcurtin> or to be clear, "Backward() is the derivative of the inputs of a particular layer with respect to the backpropagated error"

20:40 < zoq> yes, could could say that

20:40 < rcurtin> ok, just making sure---when I realized that it made the whole system a lot more clear to me, but I wasn't sure if I was correct :)

20:41 < zoq> Might be a good idea, to clarify that in the tutorial.

20:41 < zoq> Will set that on the list.

20:41 < manthan> shouldnt backward() be derivative of the backpropagated error with respect to the present layer paramters?

20:42 < manthan> so that i can backpropogate this error to previous layer

20:46 < manthan> for eg :- for ith layer, w(i)(new) = w(i)(previous) + alpha*dL/dw(i) and backpropogated error to previous layer will be dL/dw(i+1) * dw(i+1)/dwi ?

20:49 < manthan> so backward() implements logic for finding dw(i+1)/dwi given dL/dw(i+1)

20:49 < zoq> of a particular layer, in case of a ffn it's the previous one

20:49 < manthan> is this correct?^

20:51 < zoq> yes, looks correct to me

20:52 < zoq> haven't checked the backward/gradient step of the flexible ReLU layer.

20:52 < zoq> Will take a closer look at the code in the next days.

20:53 < rcurtin> the backward step looked correct to me when I did the previous review, but I am not 100% certain, only about 95% :)

20:53 < manthan> i think backward is correct but i am not sure about gradient

20:53 < manthan> this is because i am not able to clearly understand what gradient should contain

20:54 < manthan> backward is clear to me now^

20:54 < rcurtin> I think the gradient here should contain just one element, d L / d alpha

20:55 < manthan> yes it contains one element but what is error in this case which the function obtains as a function argument?

20:55 < manthan> i mean gradient() function^

21:10 < manthan> backward() - derivative of backprop error with respect to the input and gradient() - derivative of error with respect to the trainable parameter and what i wrote for backward above should be true for gradient()

21:10 < manthan> this is what the definitions look like from the various trainable layers that i saw

21:13 < manthan> with this, gradient() of flexible relu layer should always be 1 as flexible relu is (max(0,x) +a)

21:32 < manthan> rcurtin : zoq : i have updated the flexible relu gradient function now, pls have a look. the concept is clear to me now. Thanks.

21:33 < manthan> i think addition of this in the tutorial will be very useful for contributors :D