10:20 < zoq> sumedhghais: Not sure how the test framework looks like, maybe you can open a PR or something like that, so I was thinking if we need a visitor to call the specific method since we know the type?

12:38 < zoq> ironstark: Hello, do you need any help with the dlib-ml implementation? Probably a good first step would be to write an install script?

13:24 < kris1> I had a question zoq

13:24 < kris1> so in layer you use InputDataType and OutputDataType

13:26 < kris1> but i am not able to understand how did you some variables like gradient to OutputDataType.

13:26 < kris1> Is the logic that anything that gets computed within the layer is labeled as OutputDataType.

13:30 < zoq_> kris1: Yes, everything inside the layer is of type OutputDataType only the input is of type InputDataType.

13:39 < kris1> okay….. but why use InputData type and also arma::Mat<eT> when you could have replace arma::Mat<eT> by InputDataType.

13:48 < zoq> kris1: Right, in fact I already refactored some of the existing layer, to not use arma::Mat<eT> but e.g InputType for the input and OutType for the output etc. That way we could also pass e.g. arma::subview.

13:51 < lozhnikov> kris1: I am looking through your changes. It seems, you did an invalid merge. I did git reset since you committed some unnecessary code. I pointed out that here https://github.com/mlpack/mlpack/pull/1046#issuecomment-317401006

13:54 < lozhnikov> however, it seems you fixed that later

14:13 < kris1> Yes, i think we can just cherry pick the commits if we see there are some unncessary commits.

14:13 < kris1> Were you able to look at comments i gave in the PR.

14:14 < lozhnikov> I am looking through the code right now, I'll post a review soon

15:24 < kris1> lozhnikov i looked at the comments. I made some points. Were you able to look them over.

15:25 < kris1> Also one of the thing i am not sure about is that. ssRBM can outperform binary rbm on the mnist dataset. I did not see any papers that even consider using the mnist dataset. They all use “natural images”.

15:25 < kris1> for ssRBM classification testing.

15:25 < kris1> this goes for variants of ssRBM as well.

15:27 < lozhnikov> yeah, I think we should try the CIFAR dataset

15:50 < lozhnikov> kris1: I'll be unavailable this weekend

16:48 < lozhnikov> kris1: are you online?

16:48 < kris1> Yes

16:48 < lozhnikov> I'll be unavailable this weekend

16:49 < kris1> Ahhh…okay.

16:49 < lozhnikov> I'll return in the evening on Sunday

16:49 < lozhnikov> I am going to ride a bicycle for 2 days

16:50 < kris1> Wow !!! That sound exciting and tiring

16:50 < kris1> Is it part of some marathon?

16:51 < kris1> I will continue working on the weekend on ssRBM and GAN.

16:51 < lozhnikov> No, I just want to relax with my family

16:51 < kris1> I just need to clarify few things

16:52 < lozhnikov> usually, I don't participate in bicycle marathons

16:53 < kris1> 1. Gan: See my comment on Github 2. ssRBM: did you agree with slabPenalty can’t be used as a scalar i had commented upon Github

16:54 < lozhnikov> I replied at github to your comment about slabPenalty

16:54 < lozhnikov> I think it is possible to simplify expressions with slabPenalty

16:55 < lozhnikov> The second option is to use arma::diagmat()

16:56 < lozhnikov> but I think that arma::mat should work slowly in this case

16:57 < lozhnikov> regarding GAN:

16:57 < lozhnikov> I think it is possible to call optimize() only once instead of `numIterations` times

16:58 < lozhnikov> *once

16:58 < kris1> Your comment is to use arma::cumsum. So i think you mean to spike(i) * slabBias * arma::cumsum(weight.slice(i).t() * visible)

16:58 < kris1> Is that correct.

17:00 < lozhnikov> yeah, looks like that is correct, it isn't hard to verify

17:00 < kris1> okay.

17:00 < kris1> Can you explain how you would just call optimize just once for GAN.

17:01 < lozhnikov> maybe it is possible to add a counter to Gradient()

17:02 < kris1> The cause of problem for me seems to be that we have to generate the ouputs for optimisation of generator from the training of Discriminator.

17:02 < kris1> So if we have just call to the optimizer we can’t change the predictors and responses

17:03 < kris1> Also we can’t generate all the predictors and reponses in a matrix since we need to get the predictors from a trained Generator at a previous time step.

17:04 < kris1> Sorry for the 1st comment it generate oputputs for optimisation of discriminator from a trained generator.

17:05 < lozhnikov> I think you can move everything to the Gradient() function, I have to think about that

17:07 < kris1> But the evaluate function is called before the Gradient function and it utilizes the predictors as well as the responses.

17:09 < kris1> If we by pass the evaluate function. Then also it would not be possible since we have to train the generator to generate ouputs which would require us to return the gradients.

17:09 < lozhnikov> it is not hard to fix that. I think you needn't the noise matrix at all

17:10 < lozhnikov> that means you should reimplement the FFN:Gradient() function inside the GAN class

17:11 < kris1> Why woudn’t we need the noise matrix? The generator is trained on that input only.

17:11 < kris1> See point 2 above.

17:12 < lozhnikov> I think the noise matrix require some unnecessary operations

17:12 < lozhnikov> *requires

17:15 < kris1> Maybe you misunderstand me. I am saying for training of the generator at iteration i we need a generator that has been trained at iteration i -1. We generate the samples from the generator at iteration i and then call gradients function for updating it’s parameters. Sorry but i did not get your point on “noise matrix requires some unnecessary ops”

17:17 < lozhnikov> I think it is possible to generate samples inside GAN::Gradient()

17:17 < kris1> I think you can look at algorithm 1 from the goodfellow’s paper

17:18 < kris1> Aaah okay i get you point now…..

17:19 < lozhnikov> I think you can reimplement the algorithm that trains generator inside GAN::Gradient()

17:19 < kris1> Since the Gradient Function is being called n times we will train the generator and discriminator n times inside the gradient function.

17:20 < kris1> Really cool idea btw.

17:21 < kris1> Okay just the last thing should i test ssRBM on cifar 10 dataset(it’s pretty large > 1 GB). Because that is the only one that paper checks for classification accuracy.

17:22 < lozhnikov> hmm cifar-10 weights about 160MB https://www.cs.toronto.edu/~kriz/cifar.html

17:23 < kris1> That’s the pickled version.

17:24 < lozhnikov> I think we can start with that version

17:24 < kris1> okay…. i will do that then.

17:25 < kris1> I already have taken out the patches and done the preprocessing. So it would be easy.

17:25 < kris1> I think

17:27 < kris1> Thanks, Have great weekend.

17:27 < lozhnikov> I started the implementation of the mu-ssRBM, a successor of the ssRBM. It seems the mu-ssRBM solves some problems with rejection sampling. But it seems I am not in time, I'll finish that after this weekend

17:27 < lozhnikov> thanks

17:28 < kris1> Ohhh i think it already implemented here have a look https://github.com/gdesjardins/ssrbm

17:29 < lozhnikov> yeah, but I want to compare the mu-ssRBM with the ssRBM

23:14 < sumedhghaisas__> zoq: Hey Marcus. Hey Zoq... The gradients are finally correct for NTM. I will committing the code today. Next week we can do cool experiments...

23:14 < sumedhghaisas__> Just had couple of questions regarding the design.

23:15 < sumedhghaisas__> How do you think we should have the controller network right now?

23:15 < sumedhghaisas__> Currently I have hardocded the network...

23:20 < zoq> sumedhghais: Hello, with hardcoded you mean it's part of the NTM? I thought we could use a Wrapper class or something like (pass the controller to the NTM class), that way we could easily test different designs.

23:21 < zoq> Have you seen my last message about the test framework?

23:21 < zoq> sumedhghais: Not sure how the test framework looks like, maybe you can open a PR or something like that, so I was thinking if we need a visitor to call the specific method since we know the type?