09:49 < partobs-mdp> I finally understood why we have never got good results on Add and Sort: the RNN model from mlpack is not seq2seq!

09:49 < partobs-mdp> I mean, it processes sequence this way: read 1 input vector -> do calculations -> emit 1 output vector

09:49 < partobs-mdp> Proof: https://github.com/mlpack/mlpack/blob/master/src/mlpack/methods/ann/rnn_impl.hpp#L169

10:13 < zoq> partobs-mdp: For the Add task you don't need an encoder/decoder RNN, since the input and output size could be fixed. On a higher level you could use a seq2seq model, sure, but not necessarily.

14:47 < kris1> lozhnikov: I made some commits to ssRBM PR and also tested it out on the cifar dataset.

14:48 < kris1> I am not able to run the full cifar-10 dataset. My laptop just hang’s some that’s why tested it on only a small part.

14:51 < lozhnikov> kris1: I have just read your comment. sounds good

14:51 < kris1> Sorry i wasn’t able yesterday i was travelling.

14:52 < lozhnikov> no problem

14:52 < lozhnikov> how do you think, is there any sense in non-scalar slabPenalty?

14:55 < kris1> well i used a scalar slabPenalty for my experiments. And also the paper uses the values that are scalar.

14:57 < lozhnikov> In such a way I suggest to replace replace the slabPenalty matrix by a scalar. Maybe that will improve the performance, how do you think?

14:59 < kris1> I have done that in the latest commit.

14:59 < kris1> I use double for slabPenalty now.

15:00 < lozhnikov> ah, I see

15:02 < kris1> Also i mentioned that we can also convert the mu-ssrbm to ssRbm a while back. Actually i cited the wrong paper. This is the relevant paper http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6678502&tag=1

15:03 < kris1> this is the paper http://www.icml-2011.org/papers/591_icmlpaper.pdf.

15:04 < lozhnikov> I know, I have read that

15:05 < kris1> Ahh okay.

15:06 < lozhnikov> Actually, I thought about that, I started the implementation of the mu-ssRBM. I'll finish soon and I am going to try your test

15:08 < kris1> Okay. I will try to see where my GAN implmentation was going wrong tonight. If you don’t have any comments on ssRBM.

15:08 < lozhnikov> I'll look in detail in the evening

15:08 < kris1> Sure

15:58 < partobs-mdp> rcurtin: About DiscreteDistribution: I can't figure out how to use it for 1-dim discrete distribution. The unit test is not so helpful because it initializes probabilities from strings, which is clearly an overkill :)

15:59 < partobs-mdp> By the way, I got disconnected after my RNN comment. Was it responded? (Logs at mlpack.org don't help)

16:07 < zoq> partobs-mdp: Yes, http://www.mlpack.org/irc/

16:07 < zoq> partobs-mdp: For the Add task you don't need an encoder/decoder RNN, since the input and output size could be fixed. On a higher level you could use a seq2seq model, sure, but not necessarily.

16:09 < partobs-mdp> zoq: No, I mean that mlpack RNN implementation emits the next sequence element right after it read the corresponding input sequence element (hence the input/output equal sequence lengths)

16:11 < partobs-mdp> True, input size is fixed, but the way RNN class processes its input doesn't allow it to do addition - but it allows copying (because for copying it's enough to use whatever part of sequence is already in)

16:15 < zoq> The output of the Add task isn't delayed, as you did for the Copy task?

16:16 < zoq> I thought it's delayed, so the current model wout predict the input only after it has seen the complete sequence.

16:47 < partobs-mdp> zoq: Yes, that's my point - the model is asked to emit LSB of the sum after it has seen LSB of the first addend (and only it!)

16:47 < partobs-mdp> Of course, it can't be better than random guessing

16:48 < partobs-mdp> In AddTask case it could be worked around in a *horribly* ad-hoc way: feed n-th bit of *both* addends (2-dim vector) on n-th timestep.

16:49 < partobs-mdp> However, it's very ad-hoc - there should be a way to load the sequence as a whole and only then do the predictions.

16:55 < zoq> What I can think of right now, is to pad with zeros, as we did for the copy task and for the predictions we only look at the end of the sequence.

16:55 < zoq> I can do this in the next days and post the results here; I think we don't have to modify the Task classes, and we should probably move on with the model implementation. What do you think?

16:56 < partobs-mdp> Yes, I also like the idea. Also, I've read the HAM paper once again and decided that you're right about the differentiable HAM. I think I'm going to implement it, and not the hard-stochastic version.

16:57 < partobs-mdp> zoq: You mean doing this way: [0 1 0.5 1 0 0.5 0 0] -> [0 0 0 0 0 0 1 1]?

16:57 < zoq> yes

16:58 < zoq> About the differentiable HAM model, your decision; I'm fine with both ideas

23:23 < kris1> parallel sgd is failing on my local machine.

23:23 < kris1> Do i need to install open mp or some other functions for making it work

23:32 < kris1> Figured it out….i used g++7 option with cmake.

23:46 < kris1> the system dies after 40% the full build is not succesfull.