verne.freenode.net changed the topic of #mlpack to: http://www.mlpack.org/ -- We don't respond instantly... but we will respond. Give it a few minutes. Or hours. -- Channel logs: http://www.mlpack.org/irc/
travis-ci has joined #mlpack
< travis-ci>
mlpack/mlpack#239 (master - bbe9cd1 : Ryan Curtin): The build has errored.
< rsv>
Hi everyone, I'm trying to implement logistic regression via lasso using the C++ API. I've already managed to get an example working without lasso using the mlpack::regression::LogisticRegression< OptimizerType > class template. I'm confused about how to use mlpack::regression::LARS in the same way.
< naywhayare>
rsv: hi there. do you mean that you want to train a logistic regression model with an L1 and L2 penalty like LASSO?
< naywhayare>
I'm not sure I completely understand what you're trying to do
rsv has quit [Ping timeout: 246 seconds]
rsv has joined #mlpack
< rsv>
yes that's exactly what i want to do
< rsv>
but i am not sure how to use mlpack::regression::LARS in conjunction with mlpack::regression::LogisticRegression or if that's even the right idea
< naywhayare>
rsv: I don't think it will be too hard to do, but you'll have to either modify or copy some code
< naywhayare>
in src/mlpack/methods/logistic_regression/, there is a file called 'logistic_regression_function.hpp' and 'logistic_regression_function.cpp'
< naywhayare>
these two files define the LogisticRegressionFunction, which is what the optimizer uses to learn the model
< rsv>
okay
< naywhayare>
right now an L2 penalty parameter can be specified in LogisticRegressionFunction, but if you want L1 penalty, you'll have to modify the code
< rsv>
so i have to put the lambda1=0 penalty in the LogisticRegressionFunction by hand basically?
< naywhayare>
if you take a look at the Evaluate() function, you can see the objective is the loss plus a regularization term
< naywhayare>
wait, are you planning to use lambda1 = 0?
< naywhayare>
if so the code already does what you need; it has L2 regularization
< rsv>
isn't that what the L1 penalty is for lasso?
< naywhayare>
oh, right, sorry
< naywhayare>
in that case I think you are set; just use the LogisticRegression class or the logistic_regression program with lambda set to the lambda2 value you wanted to use
< rsv>
what is the default setting? l1 and l2 > 0 ?
< naywhayare>
the default setting for logistic regression is that there is no support for L1 regularization (so, l1 == 0), and L2 regularization is 0 by default but can be specified to another value
< naywhayare>
the LARS class is a regression algorithm that assumes a linear model, like Y = AX
< naywhayare>
but the LogisticRegression class is a different model
< naywhayare>
at least in the implementations here, there is not a relation between the two
< rsv>
ohhh, okay
< rsv>
this makes a lot more sense
< naywhayare>
yeah, so I think in your case, getting LASSO-like training for logistic regression just means putting the L1 penalty parameter into LogisticRegressionFunction
< rsv>
right, basically implementing something similar to the way the code already handles the optional L2 regularization
< naywhayare>
yeah
< rsv>
perfect, thanks for the help
< naywhayare>
sure, let me know if you have any issues
< naywhayare>
one thing to note is that there are two overloads of Evaluate() and Gradient()
< naywhayare>
one is meant to give the objective or gradient on the entire dataset (for optimizers like L-BFGS)
< rsv>
okay
< naywhayare>
and the other, which also takes an index of a point, is meant to give the objective or gradient on only a single point in the dataset (for optimizers like SGD)
< naywhayare>
if you take a look at the code, it should make sense
< rsv>
okay, i see that in logistic_regression_function.cpp
< naywhayare>
yeah; just something to be aware of. if you don't implement an L1 penalty in both overloads, then the L1 penalty will only work for one type of optimizer
< naywhayare>
(of course, if you only plan to use SGD, then you only need to implement the L1 penalty for the overloads that take an index :))