17:00 < alsc> so I have started thinking about the implementation of the TerminationPolicy for SGD: since classes are kind of nested but for example I am using it as a member of RMSProp…

17:01 < alsc> two questions

17:02 < alsc> 1. I was thinking of giving the full state of the Optimizer back to the user code that instantiates the termination policy, and allowing for a bool return value that decides wether computation should continue or stop… sort of bool shouldTerminate(arguments)

17:04 < alsc> this makes me think if the TerminationPolicy itself should be template over the DecomposableFunction? or?

17:07 < alsc> 2. I was thinking of using termination policy as an update policy, checking the validation error each epoch for example, and possibly changing some parameters…. so maybe instead of creating a new template we could just rethink the UpdatePolicyType concept by introducing the bool return value and passing the whole model as an argument so it can be used for crossvalidation?

17:07 < alsc> validation I meant, not crossvalidation

17:21 < alsc> zoc: I’ll also give you an overview of the changes done in the CMakeLists then..

18:37 < zoq> alsc: If you use the termination constrained as an update policy how do you integrate the update step itself, I think I miss something. I was thinking about: https://gist.github.com/zoq/4d8bce934d96240bb6fee4419f3bddf7

18:38 < alsc> zoq: yep cool I like it, you answered

18:39 < alsc> FNN<NegativeLog…> is the decomposablefunction right?

18:39 < zoq> yes, right

18:41 < alsc> neat

19:50 < rcurtin> zoq: a question about RNNs... at rnn_impl.hpp:190 (when we run Forward()), the following code is run:

19:51 < rcurtin> Forward(std::move(predictors.rows(seqNum * inputSize, (seqNum + 1) * inputSize - 1)));

19:51 < rcurtin> this gives us the time step's predictors, which we have to run on each layer

19:51 < rcurtin> but this particular code will actually do a non-contiguous copy for each time step

19:51 < rcurtin> this is because the data is organized like this:

19:52 < rcurtin> each row contains all time steps for all points sequentially; so if it's 10-dimensional data, the first 10 rows have the first time step, the second 10 rows have the second time step, and so forth ...

19:52 < rcurtin> but, if we organized the data differently, we could avoid this call to .rows() and could avoid the copy

19:53 < rcurtin> the data would need to be organized like this (if we kept it as an arma::mat):

19:53 < rcurtin> each row contains one time step for a single point; the columns are organized such that if there are N points, the first N columns are time step 0 for each point; the second N columns are time step 1 for each point; and so forth ...

19:54 < rcurtin> alternately you can represent this as an arma::cube where each slice is a time slice

19:54 < rcurtin> what do you think? I am happy to make the change (I think it will be straightforward)

19:54 < rcurtin> the only other question is how we might handle variable-length sequences cleanly (not many other toolkits have good support for that); in those cases a copy may be unavoidable, but maybe we can think about the variable-length problem some other day

20:15 < zoq> I would go with arma::mat since all of the other methods deal with arma::mat as input instead of using arma::cube, so unless you see another benefit of using arma::cube I would go with arma::mat. If you can make the adjustments happy to take alook at it afterwards.

20:15 < zoq> Also I agree, let#s deak with variable-length sequences, later.

20:17 < rcurtin> ok, I will rearrange the data accordingly and add some documentation about the data format

20:17 < rcurtin> it might take a little while to adjust all of the tests/etc. that use the RNN code

20:17 < zoq> sounds great