00:52 < uzipaz> zoq: is there a way to build Neural nets in a generic way, I using the approach used in the test code and if I want to add more hidden/dropout/dropconnect layers, I have to change the soure code and recompile... is there an alternative?

11:31 < alpha__> :uzipaz Heyy I also wanted to do something similar.. here is what I did

11:39 < alpha__> refer to this http://pastebin.com/bKKYPECQ

11:40 < alpha__> you can compile the program using g++ -std=c++11 Neural_network_using_mlpack.cpp -l mlpack -l armadillo -l boost_serialization -l boost_program_options

11:41 < alpha__> content of the cpp file given in pastebin link

11:41 < alpha__> let me know if it helps :)

17:14 < uzipaz> is the mlpack library inherently parallelized?

17:18 < zoq> uzipaz: Unfortunately no, some of the methods are parallelized using OpenMP.

17:21 < uzipaz> zoq: may I ask, what methods are parallelized?

17:24 < zoq> uzipaz: DET, and ranjan is working on a parallelized sgd optimizer.

17:27 < uzipaz> zoq: Im running FFN with 2291 input features and 842 training samples, with 2 hidden layers and the output layer following a dropout layer... it took about 6.5 hrs to train

17:28 < uzipaz> zoq: i used sgd as the optimizer with 100,000 max iterations

17:33 < zoq> uzipaz: wow, I used data sets with way more samples and features, in much less time.

17:34 < zoq> uzipaz: I would suggest, that you decrease the tolerance, but you said you set max iterations. I guess, you don't use a decent machine. Maybe you can send me your code and I'll take a look.

17:35 < zoq> * ah, I mean, I guess you use a decent machine

17:35 < uzipaz> zoq: that would be great! here is the link http://pastebin.com/3yrSm7Ar

17:36 < zoq> uzipaz: so not a raspberry pi or something like that

17:36 < uzipaz> zoq: i have core i5 laptop, with 2 physical, 4 virtual cores, each maxed out at around 2.8 ghz with 6 GB of memory

17:36 < zoq> uzipaz: okay, great

17:37 < zoq> uzipaz: Can you send me the dataset?

17:39 < uzipaz> zoq: just sent you an email

17:40 < zoq> uzipaz: Great, thanks!

17:40 < uzipaz> zoq: awesome, I look forward to hear your comments

17:48 < zoq> uzipaz: Just to be sure, you transform your complete dataset to a binary representation and not only the target values? It's uncommen to use a binary representation for the input values, so maybe there is a deeper reason behind it?

17:50 < uzipaz> i transformed to binary because all the features were nominal with 3 values each, the target class was already binary, i converted the nominal to binary to eliminate the numerical relationship between their values

17:50 < uzipaz> zoq:

17:51 < zoq> uzipaz: hm, okay

17:51 < uzipaz> zoq: though im not sure if its the best decision

17:52 < zoq> uzipaz: There are networks, that would benefit from a binary representation of the input parameter regarding runtime, because you could use bin-ops, but this is a pretty 'new' idea.

17:53 < uzipaz> zoq: you mean binary operations?

17:53 < zoq> uzipaz: yes

17:54 < uzipaz> zoq: but do have to explicitly program binary operations in our algorithm if we know the inputs are binary, shouldn't that be left for the compiler to decide?

17:54 < zoq> uzipaz: I'll have to look into the issue, but it might take some time, I'll have to do some other things first.

17:55 < uzipaz> zoq: sure, thanks for the insight, i was also thinking about using binary step function as the activation function because all the inputs are binary, i think that might increase performance

17:59 < zoq> uzipaz: If you transform everything to binary to increase the runtime you shouldn't use arma::Mat<double>, but yeah, it could increase the performance.

18:10 < uzipaz> zoq: I see, I didnt see that detail after converting to binary, thanks

18:11 < uzipaz> zoq: are you running the program with my dataset?

18:15 < zoq> uzipaz: I'll go and test the code with your dataset once I have time.

18:16 < uzipaz> zoq: I appreciate the help

21:44 < uzipaz> zoq: hey zoq, did you try running the program on my dataset i sent you?

21:46 < zoq> uzipaz: no, I can probably say more tomorrow, sorry

21:46 < uzipaz> zoq: no problem :) thanks for your time

22:22 < uzipaz> zoq: Im sry to bother you over and over again... and also its late night in germany... I just wanted to ask for any tips/pointers to make my FFN train faster... any advice will be appreciated

22:28 < zoq> uzipaz: I think, it would be a good idea not to transform the input to a binary representation, that blows up the number of input samples.

22:28 < zoq> uzipaz: Also, I would use RMSprop instead of SGD, it should converge faster. Also use a lower tolerance, maybe 0.1 is sufficient.

22:29 < uzipaz> zoq: thanks, I've never heard of RMSprop, what is it formally called?

22:30 < zoq> uzipaz: RMSprop :)

22:30 < uzipaz> zoq: is that minibatch gradient descent?

22:32 < zoq> uzipaz: No, RMSprop is an "unpublished" method proposed by Geoffrey Hinton: https://github.com/mlpack/mlpack/tree/master/src/mlpack/core/optimizers/rmsprop

22:34 < zoq> I guess this is the first reference: http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

22:36 < uzipaz> zoq: also, even after 6hr training, my NN was overfitting on training set and performing not so good on the test set... any advice to improve upon this?

22:40 < zoq> uzipaz: Yeah, in this case you should definitely decrease the tolerance value. Unfortunately there isn't a method right now to test against a test set during training. What I do in this case is to Train in batches, and use the Predict function during this training batches to check how the method performs on the test set.

22:40 < zoq> Btw. we are in the same time zone right?

22:42 < uzipaz> zoq: im in canada, I am about 4 hrs behind you

22:43 < zoq> uzipaz: The problem is, everytime the the Train function is called, it's starts by evaluating the complete training set, to get an initial error. So, what I do in this case is to comment the evaluation method in the optimizer.

22:44 < zoq> uzipaz: ah, the other Verdun

22:47 < uzipaz> zoq: so, I will try using RMSprop and increase the tolerance value, not sure what more else to try

22:47 < zoq> uzipaz: ah right, increase

22:49 < uzipaz> zoq: in RMSprop class constructor, what is the difference between argument eps and tolerence?

22:49 < zoq> uzipaz: maybe, there is some weird bug, I guess I'll figure it out tommorow or later today

22:53 < uzipaz> zoq: in RMSprop class constructor, what is the difference between argument eps and tolerence?

22:55 < zoq> eps is used for numerical stability; to avoid division by zero. tolerance is the value used to terminate the optimize before it reaches the max iterations.

22:55 < zoq> (std::abs(lastObjective - overallObjective) < tolerance) break;

22:57 < zoq> So if you use a tolerance of 0.5 and use the Predict function on the training set; you should get an accuracy of 0.5 on the training set.

22:58 < zoq> using a tolerance of 0.4 you should get an accuracy of 0.4, and so on

22:59 < zoq> It's the same parameter as for all the other optimizer.

23:00 < uzipaz> zoq: so I should leave eps as default

23:00 < zoq> uzipaz: yes

23:01 < uzipaz> zoq: and also, if I set tolerance to 0.1, i should expect accuracy on training set to be 90percent?

23:02 < zoq> uzipaz: if the optimizer terminates before it reaches the max number of iterations, yes

23:04 < uzipaz> zoq: got it :), I suppose you are gonna go to sleep soon?

23:05 < zoq> uzipaz: nah

23:09 < zoq> uzipaz: You can always write to the channel, and we get back once we have the time. In case you think, you missed something; here are the channel logs: http://mlpack.org/irc/

23:12 < uzipaz> zoq: thanks, ill keep an eye out

23:36 < zoq> uzipaz: Do you use an modified MulticlassClassificationLayer class?

23:36 < uzipaz> zoq: no

23:36 < zoq> uzipaz: okay