verne.freenode.net changed the topic of #mlpack to: http://www.mlpack.org/ -- We don't respond instantly... but we will respond. Give it a few minutes. Or hours. -- Channel logs: http://www.mlpack.org/irc/
< ShikharJ>
sumedhghaisas: You were right, Tensorflow took about 4.5 hours on multiple cores (and 11 hours on single core aggregate).
< sumedhghaisas>
ShikhaJ: wow...
< sumedhghaisas>
that big a difference?
< ShikharJ>
sumedhghaisas: Because it was utilizing upto 8 threads at a time.
< sumedhghaisas>
Maybe we should parallelize our operations as well
< rcurtin>
I suspect OpenBLAS will do a pretty good job of that, but I guess we have to see the timing to know
< sumedhghaisas>
ShikhaJ: How long does our code take currently?
< ShikharJ>
sumedhghaisas: I have only tested on single core, and that takes about 6.5 ~ 7 hours. I have just tmux'd a build with openBLAS. I'll monitor that and let you know.
< sumedhghaisas>
rcurtin: True, OpenBLAS is a library or a framework? I mean does it give a framework to parallelize or takes care of everything on its own?
< ShikharJ>
rcurtin: Yeah, OpenBLAS is running good currently, though the load average is not as high as in tensorflow, but still.
< rcurtin>
is this on one of the build systems?
< rcurtin>
ah, yeah, I see it is running on savannah
< rcurtin>
you're right, it looks like it is not parallelizing perfectly
< ShikharJ>
rcurtin: Yeah, the load average that I saw for tensorflow ranged from 2.5 to 2.9 ~ 3.0 but mostly in the lower range.
< rcurtin>
sumedhghaisas: OpenBLAS is just a BLAS replacement that uses OpenMP internally, so if you are doing big matrix multiplications it will parallelize
< rcurtin>
ShikharJ: another note (and I can kill it if you want), I just started the benchmark checkout job on that system so it builds a handful of libraries for the benchmarking system
< ShikharJ>
rcurtin: Which process is that?
< rcurtin>
actually let me kill it
< ShikharJ>
rcurtin: I'll restart the build then?
< rcurtin>
ok, stopped now... I think the effect on your runtime will be small (like 1-5% at most, probably less since it only went on for like a minute)
< rcurtin>
nah, no need, I think
< rcurtin>
another idea for parallelism---last year a different Shikhar (Bhardwaj) implemented a parallel SGD variant; it might be possible to use that here
< rcurtin>
only problem is, that worked best with objective functions where the gradient is sparse, so it might not work here
< rcurtin>
but when I think about it, I guess it should not be too hard to parallelize the FNN class... I wonder if OpenMP tasks could be used for something like this
< ShikharJ>
rcurtin: But we'll need to test on batches of size 50, to make the comparison. I'm not sure if ParallelSGD would take batch sizes into consideration.
< rcurtin>
ShikharJ: you're right, it only supports a batch size of 1 right now
< zoq>
We could save some time if we implement the EvaluateWithGradient function.
< zoq>
That way we could save another Forward pass in each iteration.
< ShikharJ>
rcurtin: From what I see on htop, big matrix multiplications are only forming a small part of the total time, so I don't think using openBLAS would be of much benefit here. The load average is barely above 1.
< rcurtin>
zoq: you're right, that could be a big savings
< rcurtin>
ShikharJ: I think you are right; as the batch size / dimensionality gets larger, the matrix multiplications will get larger also, but perhaps at this size you are right, OpenBLAS is only marginally helpful
< ShikharJ>
rcurtin: Yeah, parallelizing the FFN layer sounds like a good plan, but I'm not sure how long that would take, I'll look into zoq's idea of implementing EvaluateWithGradient Function today.
< zoq>
ShikharJ: Implementing the EvaluateWithGradient function should be straightforward, we can use the current Gradient function and return the loss.
< ShikharJ>
zoq: Yeah, do we need separate Evaluate and Gradient functions is we have a single EvaluateWithGradient? I think not but I'm not sure.
vivekp has quit [Ping timeout: 248 seconds]
< zoq>
ShikharJ: If a class doesn't implement EvaluateWithGradient it will combine the Evaluate and Gradient function, since we call the Evaluate function in the Gradient step anyway we don't need an extra Evaluate call.
< zoq>
ShikharJ: In case of the FFN class, we need both functions, since in some cases you just like to perform the forward step.
< zoq>
If EvaluateWithGradient is implemented it will prioritize it over the Evaluate/Gradient combination.
vivekp has joined #mlpack
< ShikharJ>
zoq: I see. I'll get back on this later today :)
< zoq>
ShikharJ: Great, excited to see the timings afterwards.
travis-ci has joined #mlpack
< travis-ci>
mlpack/mlpack#5215 (master - c6f9db4 : Ryan Curtin): The build has errored.
witness_ has quit [Quit: Connection closed for inactivity]
< ShikharJ>
zoq: Should I remove Evaluate and Gradient functions if I have implemented EvaluateWithGradient?
< zoq>
ShikharJ: Let us keep both for now.
< rcurtin>
yeah, some optimizers will make use of any of those three functions separately
< ShikharJ>
zoq: This was a great idea, I can already see the reduction in a number of function calls.
< zoq>
ShikharJ: This definitely effects the train time.
< zoq>
ShikharJ: So thanks for taking a closer look into this one as well.
vivekp has quit [Read error: Connection reset by peer]
< ShikharJ>
zoq: I have pushed the changes as a new commit in WGAN PR. Can you review them? Once I can ascertain they are correct, I'll tmux a build.
vivekp has joined #mlpack
< zoq>
ShikharJ: Just did a quick look, and this looks good to me, will have to check the other lines as well, but I guess this is ready to be tested.
< ShikharJ>
zoq: So do you think I should cancel the current OpenBLAS build?Or should I let it run, and then run the build with the current code to compare?
< zoq>
ShikharJ: Would be interesting to get the OpenBLAS results as well, but your choice.
< ShikharJ>
zoq: Also, do you think I can split the gan_impl.hpp file into gan_impl.hpp, wgan_impl.hpp and wgangp_impl.hpp?
< zoq>
ShikharJ: I think that is reasonable, perhaps create another folder?
< ShikharJ>
zoq: The last two would be having the evaluate, gradient and evaluatewithgradient functions of their respective classes?
< ShikharJ>
zoq: Yes that would make sense.
< zoq>
ShikharJ: Yeah, I think that would improve the readability.
< ShikharJ>
zoq: I'll let the build run, just to see how much time we can save from using OpenBLAS. Though I don't expect it to be much, maybe 20~30 minutes. But that would also give us a concrete time to improve upon.
< zoq>
ShikharJ: Agreed, even 30 minutes for free (we don't have to change anything), would be good to have.
< ShikharJ>
rcurtin: Did you spawn a benchmark job recently?
< ShikharJ>
rcurtin: There seems to be a jenkins process running, that is taking up all the cores.
< rcurtin>
ShikharJ: I thought I killed it!
< rcurtin>
hmm, the benchmark job is not doing anything
< rcurtin>
the build appears to be taking place in /home/ShikharJ/, are you sure it's not something you were running?
< ShikharJ>
rcurtin: I am running a new build now, but back then there were upto 6 jobs running under jenkins name.
< rcurtin>
I am not sure what it would have been; I restarted the benchmark checkout job, however I immediately killed it on savannah, so there should not have been any issue
< rcurtin>
let me just mark the system offline so that Jenkins doesn't try to use it
< rcurtin>
let me know when you're done with the runs and I'll bring it back online... hopefully there are no more problems, sorry about that
< ShikharJ>
rcurtin: Yeah, this build should finish in less than 6.5 hours, so after that period.
travis-ci has joined #mlpack
< travis-ci>
mlpack/mlpack#5220 (master - 255bc42 : Ryan Curtin): The build has errored.