<ShubhamAgrawal[m>
rcurtin: In ANN, why don't we store gradients and forward data in layer itself and store pointer to these allocation in FFN?
<rcurtin[m]>
that makes a huge amount of programming overhead for anyone writing a new layer, and increases opportunities for bugs (because there is more code allocating memory in different places)
<ShubhamAgrawal[m>
rcurtin[m]: Oh I am thinking about it for my proposal to change it to in-layer.
<ShubhamAgrawal[m>
Because now you need to add gradients from different layers to get a gradient for resnet or inception like network
<rcurtin[m]>
I would not support the idea of holding the memory in-layer. that's how it used to be, and there were very very many bugs (some never solved) that resulted from the complexity of that design
<rcurtin[m]>
those bugs mostly had to do with aliases that went out of date or got deallocated and so forth
<rcurtin[m]>
for your situation, I don't think it's too painful in the DAG case to simply allocate a temporary matrix for the gradient, then add it to the final gradient
<ShubhamAgrawal[m>
rcurtin[m]: Ok
<ShubhamAgrawal[m>
I can leave it as it is and change backprop algo for DAG
<rcurtin[m]>
it is true that a copy is suboptimal; but when compared with the cost of the forward and backward passes themselves, I suspect it will not be too painful
<rcurtin[m]>
(and if it is, we can find out later with a profiler and tune it as needed)
<ShubhamAgrawal[m>
Ok thanks
<rcurtin[m]>
I used to feel much more strongly about making code as fast as possible everywhere; upon reflection over the years, sometimes the cost of maintaining clever code is too high---especially if that clever code was not actually solving an empirically demonstrated performance bottleneck
<ShubhamAgrawal[m>
rcurtin[m]: Yeah
<ShubhamAgrawal[m>
I think too abstract approaches sometimes.
<fieryblade[m]>
.
<ShubhamAgrawal[m>
zoq: For writing pre-trained models proposal, do I need to write API as API is written in models wiki page?
<ShubhamAgrawal[m>
Or should I copy it to proposal too?
say4n[m] has quit [Quit: You have been kicked for being idle]
AkifEmreapolu[m] has quit [Quit: You have been kicked for being idle]
TarekNasser[m]1 has quit [Quit: You have been kicked for being idle]
manav71ManavSang has quit [Quit: You have been kicked for being idle]
Niket has joined #mlpack
Niket has quit [Ping timeout: 250 seconds]
hitesh-anandhite has quit [Quit: You have been kicked for being idle]
krushia_ has joined #mlpack
krushia has quit [*.net *.split]
<zoq[m]>
> <@shubhamag:matrix.org> zoq: For writing pre-trained models proposal, do I need to write API as API is written in models wiki page?
<zoq[m]>
> Or should I copy it to proposal too?
<zoq[m]>
If it's the same no need to copy it in again, you can reference; if it differs from the API please point that out.
<zoq[m]>
* If it's the same no need to copy it in again, you can add a reference; if it differs from the API please point that out.