07:16 < partobs-mdp> zoq: Ran experiment on the AddTask representation you described. Didn't get any sensible performance boost. The weird thing (about both representation) is that the best validation score is steadily achieved only after *two* iterations (we're talking about maxLen = 2, as always), and then SGD just grinds to a standstill

07:16 < partobs-mdp> (although there were some slight boost around 20-30 iterations)

07:17 < partobs-mdp> Oh, and I still have the issue with running the experiment in the loop

07:17 < partobs-mdp> Error message in the beginning of the second iteration: error: Mat::rows(): indices out of bounds or incorrectly used

12:03 < zoq> partobs-mdp: Hello, I'll go and rerun the experiments and post my results here. Also how can I produce the "error: Mat::rows()" issue, just run the script?

12:04 < partobs-mdp> Yes, run the script (or, rather, the bin/mlpack_lstm_baseline -t add -e 500 -b 2 -r 1 -s 128 -v --trials 5 - the script suppresses all output except the last line of cout)

12:04 < partobs-mdp> The second iteration should fail

12:05 < zoq> okay, thanks

12:05 < zoq> kris1: With model level forward you mean, a visitor which calls the Forward function not just for the layer but also for the layer inside the vector/Model()?

12:06 < kris1> Yes.

12:06 < kris1> right now the forward function for the model is private

12:09 < zoq> But what should the input be? The way it's implemented right now is that the layer that holds the vector/Model() is responsible for calling the Forward function of each layer. Maybe I misunderstood your question?

12:11 < zoq> Maybe you like to stick two FFN classes together?

12:11 < kris1> Yes, so right now the Forward function in the ffn class takes arma::mat as input. I want to directly expose this function. Right now the way it is implemented is that you call the evaluate function and then it in turn calls the forward function with predictors.col(i) as an argument

12:13 < kris1> I could actually change the Predictors matrix before calling evaluate and store the result. But that seems a but weird to me to change the predictors every time. Also i don’t need the output of outputLayer but just the output of the network.Back()

12:14 < zoq> If it helps, make them public, but maybe you could use sequential layer which is basically like the FFN class?

12:15 < kris1> okay thanks. I will take a look at the sequential class also.

12:16 < zoq> If it's easier to make the functions public, I don't mind to do that.

12:17 < zoq> ironstark: I'll take a look at the issue later today.

12:19 < zoq> partobs-mdp: "Can't see the logs - did you respond yet?" with logs you mean: http://www.mlpack.org/irc/?

13:42 < zoq> partobs-mdp: Since you changed the representation for the Add task, the network input layer should take a single element and output a single output right? So you should use RunTask<AddTask>(task, 1, 1, epochs, samples, trials); instead of RunTask<AddTask>(task, 3, 3, epochs, samples, trials);

13:44 < zoq> Also, I tested the minibatch SGD optimizer on the copy task and couldn't see any gradient explosion:

13:44 < zoq> bin/mlpack_lstm_baseline -t copy -e 800 -b 2 -l 10 -r 3 -s 1000 -v

13:44 < zoq> [INFO ] RNN::RNN(): final objective of trained model is 0.0994736.

13:44 < zoq> [INFO ] Running evaluation loop.

13:44 < zoq> [INFO ] Final score: 1

13:45 < zoq> so maybe we should switch to minibatch SGD for the other tasks as well

13:57 < zoq> I can reproduce your results

13:59 < zoq> Currently testing minibatch SGD on: bin/mlpack_lstm_baseline -t copy -e 800 -b 2 -l 10 -r 4 -s 1000 -v

13:59 < zoq> so far the object decreases in every iterations

14:18 < zoq> Also, currently we reset the update policy every time we call train; we should do: updatePolicy.Initialize(iterate.n_rows, iterate.n_cols); only once. We could introduce some parameter 'reset' that prevents this.