ChanServ changed the topic of #mlpack to: "Due to ongoing spam on freenode, we've muted unregistered users. See for more information, or also you could join #mlpack-temp and chat there."
cjlcarvalho has joined #mlpack
cjlcarvalho has quit [Ping timeout: 246 seconds]
vivekp has quit [Ping timeout: 264 seconds]
vivekp has joined #mlpack
lolbot has joined #mlpack
lolbot has quit [Ping timeout: 256 seconds]
ayesdie has joined #mlpack
ayesdie has left #mlpack []
robertohueso has joined #mlpack
cjlcarvalho has joined #mlpack
rajat_ has quit [Quit: Connection closed for inactivity]
cjlcarvalho has quit [Quit: Konversation terminated!]
cjlcarvalho has joined #mlpack
cjlcarvalho has quit [Ping timeout: 264 seconds]
caiojcarvalho has joined #mlpack
Helios has joined #mlpack
Helios has quit [Client Quit]
ImQ009 has joined #mlpack
ayesdie has joined #mlpack
blakjak888 has joined #mlpack
< blakjak888> Hi - I have a question regarding correctly building layers for dropout and batchnorm.
< blakjak888> If I want a layer of nodes with a Linear and Relu activitation to use DropOut regularization, should I add the dropout layer before Linear<> and Relu<>, or after?
< blakjak888> Same question for BatchNorm<> layer.
blakjak888 has quit []
davida has joined #mlpack
< rcurtin> blakjak888: consider the Dropout layer to be one that takes all inputs and passes only *some* of them to the output
< rcurtin> so if you want the inputs to the linear layer to be dropped out, you'd use Dropout before the layer
ayesdie has quit [Ping timeout: 256 seconds]
< davida> Hi <rcurtin>. I am blakjak888. Rejoined as I thought I was not authorised properly.
< davida> Thanks for that info, so I have been using it wrongly.
< davida> Also, for BatchNorm layer, is it the same reasoning?
< rcurtin> no problem :)
< davida> If, for some reason that I cannot think of at the moment, I wanted to add Dropout on some of the inputs, would I need to add a layer prior to my first layer?
< rcurtin> I think that BatchNorm operates in the same way, yeah
< davida> Is that the Identity Layer?
< davida> So I would add Dropout->Identity->Linear->Relu->etc?
< zoq> davida: The Identity Layer jsut forwards the output from the previous layer.
< zoq> *just
< zoq> davida: Dropout should come right after the layer: Input -> Linear -> Dropout ...
< zoq> davida: Input -> Dropout -> ... works as well.
< davida> zoq: Thanks. Can you advise me how to add a Dropout layer so I can modify the deterministic attribute before and after training? I am trying to create a variable: mlpack::ann::Dropout drp1 = mlpack::ann::Dropout<>(0.5)
< davida> but I cannot seem to add that variable with the model.Add function.
< rcurtin> ack, sorry about that, I got carried away by a conversation here in the office
< rcurtin> still involved in it... :(
< davida> e.g. using model.Add(drp1);
< zoq> davida: model.Add<Dropout<> >(0.5);
< zoq> davida: does that work for you?
< davida> zoq: it did but then I cannot access the Deterministic attribute of that layer, hence I thought I need to define a variable.
< davida> I need Deterministic() = false for training and Deterministic = true for testing
< zoq> Ah, if you use the FFN class it should be set automatically.
< zoq> But model.Add<Dropout<> >(dropoutLayer); should work as well
< zoq> where dropout Layer is: Dropout<> dropoutLayer(0.5);
< davida> OK. I am using an FFN layer but did not realise the attribute was set automatically.
< davida> How about BatchNorm layers?
< davida> Are they also after or before?
< zoq> after the actual layer
< davida> I am building a model like this: Linear->Relu->Dropout->BatchNorm->Linear->Relu->Dropout->BatchNorm-Linear->LogSoftmax
< davida> I want Dropout and Batchnorm applied to my first two hidden layers
< zoq> yeah, looks good, not sure if that's a good idea, to run Dropout before the BatchNorm but maybe it is
< davida> Thx
< davida> BTW - from Dropout() documentation I saw: "Note: During training you should set deterministic to false and during testing you should set deterministic to true." Is there somewhere it says this is taken care of in an FFN?
< zoq> Good point, we should clarify the comment.
< davida> Is it set to false by FFN::Train() and set to true by FFN::Predict() ?
< zoq> Correct, the value is set inside the FFN Evaluate method, but the value itself is set in Train/Predict.
ImQ009 has quit [Quit: Leaving]
cjlcarvalho has joined #mlpack