17:45 < blakjak888> Hi - I have a question regarding correctly building layers for dropout and batchnorm.

17:46 < blakjak888> If I want a layer of nodes with a Linear and Relu activitation to use DropOut regularization, should I add the dropout layer before Linear<> and Relu<>, or after?

17:46 < blakjak888> Same question for BatchNorm<> layer.

17:55 < rcurtin> blakjak888: consider the Dropout layer to be one that takes all inputs and passes only *some* of them to the output

17:55 < rcurtin> so if you want the inputs to the linear layer to be dropped out, you'd use Dropout before the layer

17:56 < davida> Hi <rcurtin>. I am blakjak888. Rejoined as I thought I was not authorised properly.

17:57 < davida> Thanks for that info, so I have been using it wrongly.

17:57 < davida> Also, for BatchNorm layer, is it the same reasoning?

17:59 < rcurtin> no problem :)

17:59 < davida> If, for some reason that I cannot think of at the moment, I wanted to add Dropout on some of the inputs, would I need to add a layer prior to my first layer?

17:59 < rcurtin> I think that BatchNorm operates in the same way, yeah

17:59 < davida> Is that the Identity Layer?

18:00 < davida> So I would add Dropout->Identity->Linear->Relu->etc?

18:17 < zoq> davida: The Identity Layer jsut forwards the output from the previous layer.

18:17 < zoq> *just

18:18 < zoq> davida: Dropout should come right after the layer: Input -> Linear -> Dropout ...

18:18 < zoq> davida: Input -> Dropout -> ... works as well.

18:20 < davida> zoq: Thanks. Can you advise me how to add a Dropout layer so I can modify the deterministic attribute before and after training? I am trying to create a variable: mlpack::ann::Dropout drp1 = mlpack::ann::Dropout<>(0.5)

18:20 < davida> but I cannot seem to add that variable with the model.Add function.

18:21 < rcurtin> ack, sorry about that, I got carried away by a conversation here in the office

18:21 < rcurtin> still involved in it... :(

18:22 < davida> e.g. using model.Add(drp1);

18:23 < zoq> davida: model.Add<Dropout<> >(0.5);

18:23 < zoq> davida: does that work for you?

18:23 < davida> zoq: it did but then I cannot access the Deterministic attribute of that layer, hence I thought I need to define a variable.

18:24 < davida> I need Deterministic() = false for training and Deterministic = true for testing

18:24 < zoq> Ah, if you use the FFN class it should be set automatically.

18:25 < zoq> But model.Add<Dropout<> >(dropoutLayer); should work as well

18:25 < zoq> where dropout Layer is: Dropout<> dropoutLayer(0.5);

18:25 < davida> OK. I am using an FFN layer but did not realise the attribute was set automatically.

18:26 < davida> How about BatchNorm layers?

18:26 < davida> Are they also after or before?

18:27 < zoq> after the actual layer

18:27 < davida> I am building a model like this: Linear->Relu->Dropout->BatchNorm->Linear->Relu->Dropout->BatchNorm-Linear->LogSoftmax

18:27 < davida> I want Dropout and Batchnorm applied to my first two hidden layers

18:29 < zoq> yeah, looks good, not sure if that's a good idea, to run Dropout before the BatchNorm but maybe it is

18:31 < davida> Thx

18:33 < davida> BTW - from Dropout() documentation I saw: "Note: During training you should set deterministic to false and during testing you should set deterministic to true." Is there somewhere it says this is taken care of in an FFN?

18:34 < zoq> Good point, we should clarify the comment.

18:36 < davida> Is it set to false by FFN::Train() and set to true by FFN::Predict() ?

18:38 < zoq> Correct, the value is set inside the FFN Evaluate method, but the value itself is set in Train/Predict.