16:29 <zoq[m]1> Hello, for the optimization part we rely on https://github.com/mlpack/ensmallen; ensmallen expects that a user defines a simple interface Evaluate/Gradient, what you return as a Gradient is defined by the user.

16:30 <zoq[m]1> https://ensmallen.org/docs.html#function-type-documentation has some super simple examples how the function interface looks like.

16:42 <zoq[m]1> Hm, that is an interesting problem, because you need the gradient of the image with respect to the activations.

16:42 <zoq[m]1> It's only the activation right?

16:50 <zoq[m]1> Maybe I'm missing something, but I think you don't have to change much (it will not be fast, because there are some extra copies involved). Like if you update the weights (to use the activations) before https://github.com/mlpack/mlpack/blob/45de6e11e49d430ebc069f3a991e845e0989a26d/src/mlpack/methods/ann/ffn_impl.hpp#L394 it might already do the right thing.