verne.freenode.net changed the topic of #mlpack to: http://www.mlpack.org/ -- We don't respond instantly... but we will respond. Give it a few minutes. Or hours. -- Channel logs: http://www.mlpack.org/irc/
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 244 seconds]
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas3 has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has joined #mlpack
sumedhghaisas3 has quit [Ping timeout: 276 seconds]
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas has quit [Ping timeout: 240 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
sumedhghaisas has quit [Ping timeout: 240 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 276 seconds]
govg has quit [Ping timeout: 260 seconds]
govg has joined #mlpack
witness_ has quit [Quit: Connection closed for inactivity]
witness_ has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
ImQ009 has joined #mlpack
vivekp has quit [Ping timeout: 256 seconds]
vivekp has joined #mlpack
ImQ009 has quit [Quit: Leaving]
sumedhghaisas has joined #mlpack
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
wenhao has joined #mlpack
ImQ009 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 256 seconds]
sumedhghaisas has joined #mlpack
< wenhao>
I am trying to do neighbor search with cosine distance. Does anyone how I should do that? I guess I should use NeighborSearch with IPMetric<CosineDistance> or fastmks.
< wenhao>
Does anyone know how I should do that?*
sumedhghaisas has quit [Ping timeout: 256 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 240 seconds]
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
travis-ci has joined #mlpack
< travis-ci>
manish7294/mlpack#13 (lmnn - ac54f6f : Manish): The build was fixed.
< ShikharJ>
rcurtin: It seems that tmux is not installed on savannah.mlpack.org, and I can't seem to install it using sudo, as I'm not one of the sudoers.
wenhao has quit [Quit: Page closed]
sumedhghaisas has joined #mlpack
sumedhghaisas has quit [Ping timeout: 260 seconds]
sumedhghaisas has joined #mlpack
ImQ009 has quit [Quit: Leaving]
ImQ009 has joined #mlpack
govg has quit [Quit: leaving]
ImQ009 has quit [Read error: Connection reset by peer]
ImQ009 has joined #mlpack
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 276 seconds]
< ShikharJ>
zoq: Since, the work on standard GAN is nearing completion, I'll start working on DCGAN implementation and testing for the rest of Phase I (~1.5 weeks)
sumedhghaisas has joined #mlpack
< ShikharJ>
zoq: I also opened a PR for shed_cols and shed_rows in Cubes for armadillo.
sumedhghaisas2 has quit [Ping timeout: 244 seconds]
sumedhghaisas has quit [Ping timeout: 244 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 256 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 255 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 240 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 240 seconds]
sumedhghaisas2 has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas3 has joined #mlpack
< rcurtin>
ShikharJ: fixed, sorry about that
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas3 has quit [Ping timeout: 255 seconds]
sumedhghaisas2 has joined #mlpack
sumedhghaisas has quit [Ping timeout: 255 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas3 has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 276 seconds]
sumedhghaisas has quit [Ping timeout: 276 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas3 has quit [Ping timeout: 244 seconds]
sumedhghaisas has quit [Ping timeout: 276 seconds]
manish7294 has joined #mlpack
sumedhghaisas2 has joined #mlpack
< manish7294>
rcurtin: Currently I have written a custom knn accuracy calculator. Can you please check it? https://pastebin.com/MFdVDcrz
< Atharva>
sumedhghaisas2: You there?
sumedhghaisas2 has quit [Quit: Yaaic - Yet another Android IRC client - http://www.yaaic.org]
sumedhghaisas has joined #mlpack
< sumedhghaisas>
Atharva: Hi Atharva
< Atharva>
Should we begin?
< sumedhghaisas>
Sure thing.
< sumedhghaisas>
Hows it going so far?
< Atharva>
So, I read your mail . Do you mean to say that if we embed a linear layer in the sampling layer, we don't give the option to the users to use other types of layers to generate mean and std?
< Atharva>
Whatever implementation I could find on the internet used only linear layers to output the mean and std
< sumedhghaisas>
Yes. I am not yet sure what other type of layers the users might want to use. Although I do have one use case in mind
< Atharva>
It is going good, I am having fun but am stuck at some parts
< Atharva>
which case?
< sumedhghaisas>
In the current work I am doing, the model restricts the mean to be in certain range, lets say from [-x, x]
< sumedhghaisas>
To emulate this I add a non-linearity over the linear layer before reparametrization
< sumedhghaisas>
but I understand it is a very specific case
< Atharva>
Oh, I understand.
< Atharva>
Yeah it is, because then the users would have to output the right size of the matrix.
< sumedhghaisas>
But in general, I am more inclined towards better control and higher responsibility designs. So in this case we could also take Ryan opinion
< sumedhghaisas>
@rcurtin
< Atharva>
For people new to VAEs, we would have to throw a very detailed error about what to do
< sumedhghaisas>
yes I agree, and I am not sure if we can throw such an error
< sumedhghaisas>
lets keep this at the back of our head, and ponder over it
< sumedhghaisas>
for now we will proceed by embedding the linear layer
< sumedhghaisas>
in case we decide to shift, it would be a minor PR
< sumedhghaisas>
what you think?
< Atharva>
I think that will work
< Atharva>
I can later make the changes
< sumedhghaisas>
Sounds good.
< Atharva>
That's almost done then, I am just stuck on the delta function.
< sumedhghaisas>
Now did you get a chance to replace the weights and bias with Linear<> layer? I think that will solve all the derivative problems
< Atharva>
I thought I had it yesterday, but then I tried implementing it and the differentiation turned out to be wrong
< sumedhghaisas>
ahh sorry. I was talking about a different issue I guess.
< sumedhghaisas>
which delta function?
wenhao has joined #mlpack
< Atharva>
Delta function in the implementation of the sampling layer.
< Atharva>
The thing is, we have to differentiate the final output which is : mean + std % gaussian sample w.r.t. the mean and the std
mikeling has joined #mlpack
< Atharva>
and then join those two delta matrices to get out final delta
< sumedhghaisas>
ahh you mean the first derivative, I see, I confused it with kronecker delta
< sumedhghaisas>
umm... Did you read my comment regarding replacing the weights and bias with Linear<> in the mail I sent?
< Atharva>
Yeah, so even after that we will need to add some gradients, right?
< Atharva>
beacuse we use the outputs of that linear layer to calculate the final sample
< sumedhghaisas>
indeed, but then the equation becomes simpler, sorry I got confused by the 'weight' in your equations
< sumedhghaisas>
so the equation is - output = mean + sample .* stddev
< Atharva>
no problem
< rcurtin>
sumedhghaisas: keeping in mind that I am not familiar with GANs very much, my opinion would usually be that allowing extra flexibility is good if it's possible
< rcurtin>
from a high level it doesn't seem like it would be hugely difficult to allow different layer types for a sampling layer, but again, I am not an expert in GANs, so my input may not be 100% useful :)
< sumedhghaisas>
so d(output) = sample .* d(stddev)
< sumedhghaisas>
this way we could backpropage the error of stddev and mean
< Atharva>
yeah and d(output) = d(mean), right? so there will be two gradients, w.r.t. std and mean respectively
< sumedhghaisas>
am I making any sense? :)
< Atharva>
Yeah I got it :)
< sumedhghaisas>
rcurtin: I agree. I always prefer flexibility :) I think if we document enough it should be pretty clear, also person using VAEs will be familiar with such architectures
< Atharva>
sumedhghaisas: So, should I not embed the linear layer?
< sumedhghaisas>
Atharva: Also I think we need to bound the stddev with a Softmax non-linearity
< Atharva>
yeah, I got that from the mail :)
< Atharva>
But why not leaky relu?
< sumedhghaisas>
Atharva: Yes I think that would be better, so the sampling layer (I would like to call it Reparametrization Layer, as its more common terminology in the literature) will only take a vector of 2n , break it in 2 parts
< sumedhghaisas>
first part would be mean, and the second would be bounded with Softmax
< rcurtin>
ack sorry you are doing VAEs not GANs. oops :(
< Atharva>
rcurtin: It's okay :)
< sumedhghaisas>
rcurtin: but I think flexibility is universal :)
< sumedhghaisas>
Atharva: Also this way there won't be any trainable parameters in the layer, so we only do backward
< sumedhghaisas>
Atharva: oops... seems like we don't have Softplus non-linearity :(
< Atharva>
About that, why not leaky relu?
< Atharva>
It will solve the problem of negative values but won't change the function much
< sumedhghaisas>
hmmm... that oes sound like a viable alternative to me, I think leaky ReLU is defined in such a way to provide some small gradient even when the unit is not active
< sumedhghaisas>
that would help for sure
< sumedhghaisas>
although let me think it over for a moment
< Atharva>
Okay, sure. I also have some other things to discuss after this.
< sumedhghaisas>
Atharva: wait, I just googled the Leaky relu function, its max(x, ax) right?
< sumedhghaisas>
thats not strictly positive... or am I missing something?
< Atharva>
oh sorry
< Atharva>
I was thinking about relu but with some small value added to it so that it's always a little more than zero
< rcurtin>
manish7294: the KNN accuracy calculator looks just fine to me
< sumedhghaisas>
I think ReLU will cause a problem here... if some initialization turns the unit off (produces negative value) it will create a problem in training that latent, given that the initialization is usually done with clipped uniform normal
< Atharva>
Okay then, we will go with softplus, but you said it isn't in the codebase
< sumedhghaisas>
the other option is e^(output)
< sumedhghaisas>
although this function suffers from local optima is KL divergence
< Atharva>
Yeah, so we just train ln(std) instead
< sumedhghaisas>
ln(std)? that won't be strictly positive, will it? if the value is under e
< Atharva>
Sorry, I meant to say that we output ln(std) and then e^(output) to get std. I was just elaborating what you said.
< sumedhghaisas>
ahh and by local optima I mean, with some initialization the output of the layer becomes too high, as e is raised by it
< Atharva>
I see, what do you suggest we do?
< sumedhghaisas>
in this bowl of locality the KL is huge, thus network is forced to optimize the log likelihood only
< Atharva>
That's defeats the purpose of adding KL divergence :|
< sumedhghaisas>
hmm... it depends on our timeframe
< sumedhghaisas>
for now, lets go ahead with e^(output)
< Atharva>
Okay
< sumedhghaisas>
Softplus is not a difficult thing to add, the function is log(exp(features) + 1)
< sumedhghaisas>
It would be ideal to add that at later stage and replace the e^(feature) function
< sumedhghaisas>
as we are running close to our schedule, I think we should focus on the timeline for now
< Atharva>
Should I just use the softplus directly in the layer for now, instead of adding it to the codebase
< sumedhghaisas>
umm... if we put that much effort in implementing the function and taking the derivatives, it would take 5 more minutes to add that in a separate file under 'layer' module :)
< Atharva>
That's true, so for now I will just use e^()
< sumedhghaisas>
its just copy pasting the forward and backward function lines, and adding that class in layers.hpp
< sumedhghaisas>
yes that sounds good :)
< Atharva>
About the KL divergence now, I didn't know that a PR for that has already been merged into the codebase
< sumedhghaisas>
huh... I am not sure I got the notification for that PR. Did you mention me in that PR?
< sumedhghaisas>
i don't understand why create a separate API for KL divergence
< sumedhghaisas>
why not use the current 'dist' module in core
< sumedhghaisas>
Atharva: it is merged though
< sumedhghaisas>
hmm... I will take a look
< Atharva>
Sorry, I didn't understand how we use the dist module for KL divergence.
< sumedhghaisas>
although what do you mean by formula used for the functions and not for the parameters?
< sumedhghaisas>
ahh okay, so KL divergence is basically just a distance between one distribution from another, so implement a KL function in any distribution class, which accepts an another distribution class to produce KL
< Atharva>
arma::accu(input % (arma::log(input) - arma::log(target))); I may be wrong but how do we calculate KL divergence between two distributions with this?
< sumedhghaisas>
I see what they have done. They have implemented the actual definition of KL divergence
< Atharva>
Yes, that's what I meant to say.
< Atharva>
Is it even possible to have a genral expression for KL divergence which can be used for kinds of distributions?
< sumedhghaisas>
hmm... so they are evaluating the KL empirically for a batch.
< sumedhghaisas>
I don't think it will be useful for us
< sumedhghaisas>
we have well defined distributions
< Atharva>
Yeah
< Atharva>
How should we proceed then?
< sumedhghaisas>
So we proceed by our 'dist' API then, we implement the KL and its derivative in 'dist'
< sumedhghaisas>
let me try to elaborate on that
< sumedhghaisas>
So the function will accept an object with template parameter
< sumedhghaisas>
we can specialize the template to accommodate different implementation of the KL
< sumedhghaisas>
so in Gaissian we will have a template specialization which accepts other Gaissian dist objects and returns a KL but for other dists its will throw a compile time error
< sumedhghaisas>
hope I am not confusing you too much
< Atharva>
No, I understand, it will become clearer when I actually implement it.
< sumedhghaisas>
This way someone else could extend this API to support KL between various other combinations for his own purpose
< sumedhghaisas>
We will only implement the KL between 2 gaussians
< Atharva>
roger that :)
< sumedhghaisas>
sounds good :) Now we just a good name for the function and its derivative function :)
< sumedhghaisas>
'kl' sounds good for forward, what should be backward?
< Atharva>
klBackward?
< sumedhghaisas>
'kl_backward'? or 'kl_derivative' ?
< sumedhghaisas>
you are right... 'kl_backward' sounds better
< sumedhghaisas>
Atharva: now that we are using 'dist' API... there is one little request I would like to make
< Atharva>
yeah, in that, when going forward kl comes from the latent variables, but when going backward it goes to the last layer of the vae network, how do we handle that?
< Atharva>
wont the derivatives change?
< sumedhghaisas>
ahh yes...
< sumedhghaisas>
we need to make sure we accommodate the KL loss in final loss
< sumedhghaisas>
ahh wait... but the KL loss does not affect the decoder in VAE
< sumedhghaisas>
does it?
< sumedhghaisas>
so the original architecture should work just fine...
< Atharva>
I think it won't but the derivative will have to be w.r.t. the decoder
< sumedhghaisas>
Let me see if I can explain this in a better way
< sumedhghaisas>
So in our current framework we have a final loss, for which we will implement log likelihood loss
< sumedhghaisas>
the error of that loss will propagate through the decoder layers and update the weights
< sumedhghaisas>
when it will reach the repar layer, we will update the error with the error from KL loss and backpropagate it through the encoder layers
< sumedhghaisas>
if you do the math, KL loss acts as a constant for decoder layers
< sumedhghaisas>
the derivative will only play a role when mean and stddev is involved
< sumedhghaisas>
maybe I am wrong but let me check this again
< sumedhghaisas>
is it clear my point?
< Atharva>
Yeah it is, but won't the derivative just go to zero for KL loss then?
< sumedhghaisas>
I am not sure why they will go to zero
kevgeo has joined #mlpack
< sumedhghaisas>
the error for mean, will be basically the sum of error from upper layer and error from the KL loss term
< Atharva>
Because as you said, KL loss acts as a constant for decoder layers, and the derivative of the total loss will be w.r.t. to the decoder
kevgeo has quit [Client Quit]
< sumedhghaisas>
ahh maybe I see your confusion
< sumedhghaisas>
the derivative of KL loss will be zero for decoder layers
< sumedhghaisas>
but they won't be zero for mean and stddev in repar layer
< sumedhghaisas>
so we will update the error in that layer
< Atharva>
understood, sorry if i am requiring too much help. I think it will get easier when I start implementing the vae class and these prerequisites are taken care of
< sumedhghaisas>
no problem dude :) Backpropagation is sometimes little tricky... Although I would like to suggest a simple exercise for you if you want to clear these concepts
< Atharva>
That would be very helpful, what is the exercise?
< sumedhghaisas>
So write down a neural network on paper with 1 feed forward layer and non linearity given by function 'g' as an encoder
< sumedhghaisas>
the output goes directly to the repar and then for decoder same feed forward layer with same 'g' function as non linearity
< sumedhghaisas>
so in total there are 3 layers
< sumedhghaisas>
now on paper write down the errors propagating backwards for each layer based on the loss function
< sumedhghaisas>
Also write down the weight updates
< sumedhghaisas>
there are total 3 errors flowing backwards... so it should be very quick to calculate those
< Atharva>
I will do that surely.
< sumedhghaisas>
once you have those 3 errors written down... just analyze them
< sumedhghaisas>
its a fairly quick exercise I used to do in college :)
< Atharva>
Okay :)
< Atharva>
So, I think I will start implementing all that we discussed.
< Atharva>
I will try to do it by sunday night, so that from Monday I can start with VAE class as planned
< sumedhghaisas>
Atharva: Sorry for taking too much of your time and also sorry to be very pedantic about this, but could we also shift the sampling to 'dist' , its a fairly easy copy paste. :)
< sumedhghaisas>
what I mean is... The repar layer receives, vector of 2n, which we split it in 2 vectors of n
< jenkins-mlpack>
* Marcus Edel: Adjust parameter size and use alias inside conv operation.
< jenkins-mlpack>
* sshekhar.special: Fixed the scope of power function and typo in tests
< jenkins-mlpack>
* sshekhar.special: Added name to contributors
< jenkins-mlpack>
* sshekhar.special: Fix Continuous Mountain Car environment
< jenkins-mlpack>
* Ryan Curtin: Fix spelling (thanks Manish!).
< rcurtin>
looks like there may be a memory leak somewhere, occasionally the tests are hanging
sumedhghaisas has quit [Ping timeout: 255 seconds]
sumedhghaisas has joined #mlpack
sumedhghaisas2 has quit [Ping timeout: 256 seconds]
< manish7294>
rcurtin: It looks like slake doesn't have tmux too.
witness_ has quit [Quit: Connection closed for inactivity]
< rcurtin>
manish7294: hang on, let me get it there also
< manish7294>
rcurtin: great :)
< rcurtin>
manish7294: ok, installed. don't hesitate to ask if there are any other missing packages :)
< manish7294>
rcurtin: sure :)
sumedhghaisas has quit [Ping timeout: 256 seconds]
sumedhghaisas has joined #mlpack
< ShikharJ>
zoq: Are you there?
< ShikharJ>
zoq: Since the standard GAN and DCGAN only differ in the layers they employ (DCGAN makes use of transposed convolutions in place of Bilinear Interpolation, and a few minor changes), I was thinking to make use of the existing code for DCGAN as well.
< ShikharJ>
zoq: That way, when we extend support for batch sizes, we'll automatically have the support in both of them, and if we were to try something additional, such as separating the optimizer for discriminator and generator, we'll get the benefits in both.
sumedhghaisas has quit [Ping timeout: 256 seconds]
sumedhghaisas2 has joined #mlpack
< ShikharJ>
zoq: lozhnikov: Let me know what you think of that?
< ShikharJ>
zoq: We'll probably need to have a separate module in the case for Wasserstein GAN, but that's a problem for later.
travis-ci has joined #mlpack
< travis-ci>
ShikharJ/mlpack#167 (master - 949ab83 : Ryan Curtin): The build has errored.
< rcurtin>
I'm running valgrind on mlpack_test but it takes a really long time... might be a while before I'm able to isolate any problem that is causing tests to hang