verne.freenode.net changed the topic of #mlpack to: http://www.mlpack.org/ -- We don't respond instantly... but we will respond. Give it a few minutes. Or hours. -- Channel logs: http://www.mlpack.org/irc/
chenzhe has quit [Ping timeout: 246 seconds]
trapz has quit [Quit: trapz]
Alvis__ has joined #mlpack
Alvis_ has quit [Ping timeout: 246 seconds]
Alvis has joined #mlpack
aashay has quit [Quit: Connection closed for inactivity]
Alvis__ has quit [Ping timeout: 246 seconds]
Alvis has quit [Ping timeout: 246 seconds]
Alvis has joined #mlpack
< wiking>
rcurtin, y0
trapz has joined #mlpack
trapz has quit [Quit: trapz]
< rcurtin>
wiking: hello there
< wiking>
hihi
< wiking>
what's your count this year?
< rcurtin>
of applications?
< rcurtin>
69
< wiking>
niiice
< wiking>
:)
< rcurtin>
a lot to look through :(
< rcurtin>
last year it was 119...
< rcurtin>
how many did you get?
< wiking>
38
< rcurtin>
a more manageable count :)
< wiking>
i just went through of them
< wiking>
16 ignore
< wiking>
:)
< rcurtin>
I haven't had a chance yet to really look through ours
< rcurtin>
there were a bunch of emails of the type 'please look through my proposal and give comments' but I thought the deadline was 12 hours after it actually was... so I did not get to those in time... :(
< wiking>
:>
< wiking>
yeah i mean i'm not so sure i understand the rational
< wiking>
with these last minute application
< wiking>
some students really took their time and effort to put together a draft like 1 month before the deadline
< rcurtin>
I guess the stipend is sufficiently high that people think "why not give it a shot"
< wiking>
that manifested in a good application actually
< wiking>
ah but you know
< rcurtin>
same here, a lot of people have been in touch for many weeks before the deadline and all of them are quite well prepared and have made nice patches, etc.
trapz has joined #mlpack
trapz has quit [Quit: trapz]
vinayakvivek has joined #mlpack
Alvis has quit [Ping timeout: 246 seconds]
Alvis has joined #mlpack
diehumblex has joined #mlpack
vss has joined #mlpack
vss has left #mlpack []
chenzhe has joined #mlpack
Alvis has quit [Ping timeout: 246 seconds]
yannis has joined #mlpack
yannis is now known as Guest95911
chenzhe has quit [Ping timeout: 260 seconds]
kdkw has quit [Ping timeout: 260 seconds]
vinayakvivek has quit [Quit: Connection closed for inactivity]
witness_ has joined #mlpack
govg has quit [Ping timeout: 240 seconds]
chenzhe has joined #mlpack
Guest95911 has quit [Quit: Guest95911]
yannis has joined #mlpack
yannis is now known as Guest95911
Guest95911 is now known as yannis
yannis is now known as Guest62259
chenzhe has quit [Ping timeout: 256 seconds]
vss has joined #mlpack
Guest62259 has left #mlpack []
< vss>
rcurtin : Atleast its over for now :3
govg has joined #mlpack
govg has quit [Ping timeout: 260 seconds]
witness_ has quit [Quit: Connection closed for inactivity]
vivekp has quit [Ping timeout: 268 seconds]
vivekp has joined #mlpack
Trion has joined #mlpack
vss has quit [Ping timeout: 260 seconds]
govg has joined #mlpack
govg has quit [Ping timeout: 246 seconds]
Alvis has joined #mlpack
aashay has joined #mlpack
< Trion>
when model has been training for hours and error happens http://gph.is/1hFGH5a
aashay has quit [Quit: Connection closed for inactivity]
Alvis__ has joined #mlpack
Alvis_ has quit [Ping timeout: 246 seconds]
Alvis__ has quit [Ping timeout: 246 seconds]
trapz has joined #mlpack
Alvis has joined #mlpack
Alvis has quit [Ping timeout: 246 seconds]
govg has joined #mlpack
trapz has quit [Quit: trapz]
trapz has joined #mlpack
< rcurtin>
zoq: any more comments about SMORMS3 (#899) or should I go ahead and merge it? I think it looks fine now
< cult->
what methods are available to measure how big is the Hoeffding tree? i want to somehow verify its robustness before i apply Classify().
< cult->
i can see the methods, like NumChildren() MinSamples(), MaxSamples() but i don't know which one to use
< cult->
what is MajorityClass()?
< cult->
or how should I approach this problem? can I measure on how many observations the tree has been already trained? or how many times?
< cult->
s/measure/request
< rcurtin>
so you could use NumChildren() recursively but that might be a bit irritating
< rcurtin>
here is a simple code snippet, I think it will work:
< rcurtin>
std::stack<HoeffdingTree<>*> stack;
< rcurtin>
stack.push(&tree); // 'tree' is the root of the tree
< rcurtin>
size_t nodes = 0;
< rcurtin>
while (!stack.empty())
< rcurtin>
{
< rcurtin>
HoeffdingTree<>* node = stack.top();
< rcurtin>
stack.pop();
< rcurtin>
nodes += node.NumChildren();
< rcurtin>
for (size_t i = 0; i < node.NumChildren(); ++i)
< rcurtin>
stack.push(&node.Child(i));
< rcurtin>
}
< rcurtin>
MajorityClass() is the class most often seen by that particular node during training
< rcurtin>
so if you asked for the predicted class for some sample, the tree would percolate that sample to the appropriate leaf node and then the predicted class would be MajorityClass() in that leaf node
< rcurtin>
I should add a NumDescendants() function to HoeffdingTree<>, let me open an issue for that since I am doing other things right now
< cult->
yeah, that would be nice
< cult->
but the NumChildren if for the top nodes for the tree? or all the children across the tree?
< cult->
s/if/is
< rcurtin>
the NumChildren() gives the count of the direct children of a node in the tree, not a count of all of the nodes in the tree
< rcurtin>
hence the need to recurse through the tree like the stack solution I gave above
< zoq>
rcurtin: I think SMORMS3, RMSprop and Adam/AdaMax is ready.
< cult->
MajorityPRobability() is something like 0.5 or 50.0 ?
< cult->
but its only for one leaf node
< cult->
?
chenzhe has joined #mlpack
< rcurtin>
zoq: ok, I'll merge it
< rcurtin>
cult-: it's in [0, 1] not [0, 100]
< rcurtin>
and it is the probability of the majority class, but only for that node
< rcurtin>
note that every node in a Hoeffding tree is of type HoeffdingTree<>
< rcurtin>
so every node, even the root and high-level nodes, implements the same interface
< rcurtin>
so if you call MajorityClass() on the root of the tree, what it will give you is the majority class of the entire dataset, and MajorityProbability() will give you the percentage of points (in [0, 1]) belonging to the majority class
< cult->
ok now its clear, thanks!
< cult->
until the feature request is done i should do something like if (MajorityProbability() > 0.2) {}
< cult->
question is what would be considered an appropriate threshold value
< cult->
but thats very data specific i guess
< cult->
hm, rather root.MajorityProbability() < 0.9
Alvis has joined #mlpack
< rcurtin>
cult-: I don't understand, what do you mean to do with that code?
sumedhghaisas has joined #mlpack
< cult->
rcurtin: to make sure that the training set is separable and not dominated by one class only
trapz has quit [Quit: trapz]
travis-ci has joined #mlpack
< travis-ci>
mlpack/mlpack#2248 (master - 0242c23 : Ryan Curtin): The build is still failing.
< sumedhghaisas>
zoq: Hey Marcus, have you already thought about the regularization framework?
< zoq>
sumedhghais: What you described in section 2 "Batch Normalization"?
< sumedhghaisas>
zoq: ohh umm... no I mean like if I want to add l2 regularization on weights of certain layer...
< zoq>
ah, okay ... no I haven't thought about that ...
< sumedhghaisas>
I was thinking maybe ... somehow with iterators we can create functions to return the weights.
< sumedhghaisas>
and add L2 regularization layer
< sumedhghaisas>
but the more I think about it... the more complicated it gets
< sumedhghaisas>
somehow the gradient function of the layer has to be changed
< rcurtin>
sumedhghaisas: I hate it when that happens, it seems like that is every day for me :)
< zoq>
hm, I have to think about it ... right now I can't think of an easy solution
< zoq>
if you can think of a proof of concept, eager to take a look
< sumedhghaisas>
rcurtin: haha... I do have a solution but I really hope that there is a better one... or else I will lose faith in life :P
< sumedhghaisas>
zoq: proof of concept? sorry I didnt you
< zoq>
Maybe you can think of something that could work, but it's not "perfect" as it should be?
< sumedhghaisas>
zoq: ahh okay... I do have that. but try not to judge :P
< sumedhghaisas>
okay so first I though that 'model.Add<Linear<> >(trainData.n_rows, hiddenLayerSize)' should return the parameters of that layer...
< sumedhghaisas>
So that the user has access to add regularization over it
< sumedhghaisas>
first problem solved...
< sumedhghaisas>
now the parameter... rather than being a matrix
< sumedhghaisas>
be an object ... which stores list of objects each associated with the extra term of that weight in the error function
< sumedhghaisas>
So for example...
< sumedhghaisas>
weight matrix will be a object which stores a matrix and an empty list
< sumedhghaisas>
now when that weight is returned... I apply ApplyL2Regularization(weights)
< sumedhghaisas>
it adds an object to that weight object associated with the error gradient of that weight in the loss function
< sumedhghaisas>
now in the gradient function ... all we have to do ... is call all the gradient functions in the list that parameter holds
< sumedhghaisas>
But indeed the better implementation is the generalization of this
< sumedhghaisas>
so all the parameters of the model should be separate than the layers ... they should be stored in a master object
< sumedhghaisas>
now when every layer is instantiated... the layer will add the parameter objects in the master object and add its gradient wrt itself...
< zoq>
I mean the parameters are basically separated from each layer, and there is this master parameters matrix.
< sumedhghaisas>
yes... but accessing an individual parameter and applying regularization over it will be difficult from it...
< zoq>
I agree
< sumedhghaisas>
what I propose is... create a master storage of all weights. Each layer will create parameter there and store
< sumedhghaisas>
its reference
< sumedhghaisas>
it will also return the reference of those weights to the user...
< sumedhghaisas>
layer will simply add the gradient of that parameter with respect to itself...
< sumedhghaisas>
if each layer does this... its a most generic architecture
< sumedhghaisas>
what do you think?
< zoq>
hm, model.Parameters() is the weight storage, and you could access the parameters also via model.Model()[0].Parameters() which is a reference to the weight storage for a specific layer.
< zoq>
same for the gradients
< sumedhghaisas>
I agree... I also thought of that. But the problem remains is that how to add extra gradient terms while computing the gradient with respect to any parameter?
< sumedhghaisas>
this is due to the fact that the gradient is handled solely by the layer that creates it
< sumedhghaisas>
but in a general structure of any neural network its not the case
< sumedhghaisas>
the gradient is overall with respect to the loss function ... and can be due to multiple layers...
< sumedhghaisas>
This also solves the problem of sharing weights ... and any other model that may share parameters between them
< zoq>
There is a problem with sharing weights?
< sumedhghaisas>
ahh wait.. I see what you mean now
< zoq>
So, you propose something like an execution graph, right?
< sumedhghaisas>
yes... thats what I thought. But I realized thats Tensorflow :P
< zoq>
:)
< sumedhghaisas>
But I still do think that execution graph will be much better
< zoq>
This is definitely an interesting problem, and I have to think about it.
< sumedhghaisas>
but if we keep returning the references and adding them to different layers
< sumedhghaisas>
it is an execution graph... just not explicit
vinayakvivek has quit [Quit: Connection closed for inactivity]
< sumedhghaisas>
also Marcus... you are also interested in Reinforcement Learning right? If you have some time, I am building an actor only learning algorithm using MCTS sampling. Would love to discuss it with you
< zoq>
Sounds interesting, I have to finish some other things first, so if you like we can talk about the idea in the next days, tomorrow?
< sumedhghaisas>
Sure. Tomorrow sounds perfect. Are you usually online around this time?
< zoq>
yes, we are almost in the same timezone right? +-1h