00:39 < zoq> rcurtin: In your bandicoot experiments did you look into cooperative groups? I did a first quick test and it looks like it's faster compared to the reduce kernel, I'm not using the exact same kernel that is in bandicoot right now, will do that next.

00:39 < zoq> But maybe you already did a quick test.

00:40 < rcurtin> not at all, I don't know much about cooperative groups

00:40 < rcurtin> do you mean that it's faster than the accu() kernel I wrote? if so awesome!

00:41 < zoq> Maybe, I'm not sure nvprof shows the correct numbers.

00:42 < rcurtin> I guess you could also test with the benchmarks/accu.cpp program to see

00:42 < rcurtin> I had a lot of fun tuning the accu kernel, but I didn't use any CUDA features that were super advanced (mostly because I am not *that* advanced at CUDA :))

00:42 < zoq> Yeah, it's quite a lot of fun :)

00:43 < zoq> 0.01% 76.185us 3 25.395us 7.1040us 58.421us cuLaunchKernel

00:43 < zoq> that's the result for the bandicoot accu kernel

00:43 < zoq> Time(%) Time Calls Avg Min Max Name

00:44 < rcurtin> did the output come through right? I only see the header and one row for cuLaunchKernel

00:44 < rcurtin> (and they came through out of order, at least here)

00:44 < zoq> Yeah, I posted the header afterwards.

00:45 < zoq> Now I can't find the result for the other kernel ..

00:45 < zoq> 0.01% 15.359us 1 15.359us 15.359us 15.359us cudaLaunchKernel

00:46 < zoq> that's the one that uses Cooperative Groups

00:46 < rcurtin> nice, although it only ran once, vs. the accu kernel that ran 3 times

00:46 < rcurtin> the min for the accu kernel is actually less than cooperative groups, but I wonder if that is just noise

00:46 < rcurtin> certainly the average looks way better for the cooperative groups kernel

00:46 < zoq> the accu kernel has 3 kernel calls, I think

00:47 < zoq> the device function is included

00:47 < rcurtin> oh, good point

00:47 < rcurtin> I think at least one of those calls is just getting the result element from the GPU

00:49 < zoq> I think it's accu, accu_warp_reduce and whatever the name of the device function is

00:49 < rcurtin> ah, ok

00:49 < zoq> Will integrate the kernel and use the benchmark script.

00:50 < rcurtin> yeah, awesome if you can get such a huge speedup in addition to what's already there

00:50 < rcurtin> how big is the matrix you are accu()'ing in those numbers?

00:51 < zoq> I use the same settings as for the benchmark script.

00:51 < zoq> 1000000000

00:51 < rcurtin> wow, am I reading that wrong? 1 billion elements in 15 microseconds?

00:51 < zoq> The numbers are strange.

00:52 < rcurtin> well, if they are *relatively* correct (i.e. the new kernel *is* that much faster than the old one), I guess it doesn't matter, it is still more speedup :)

00:52 < zoq> True :)

00:57 < rcurtin> I think the Julia numbers I showed in the bandicoot issue were incorrect---I had a small bug in the objective computation that meant it did a little less work than it needed to

00:59 < rcurtin> (I don't think it's a huge difference though)

01:01 < zoq> Good, it's the main reason why I look into the accu kernel :)

01:01 < rcurtin> well, I still think there will be a decent speed difference, so some acceleration will still be necessary, I just don't know quite how much :)

01:01 < rcurtin> I may still have a bug in my code somewhere---I'm still debugging some strange results

01:02 < rcurtin> but it appears that for the runs that I *do* have working, MNIST with 50 epochs with a batch size of 128, Julia takes ~9.1s, bandicoot+CUDA takes ~14.3s

01:02 < rcurtin> (both converge to the same accuracy)

01:02 < zoq> Didn't know Julia had such a good GPU acceleration.

01:03 < rcurtin> yeah, my guess is that they are just calling kernels with no overhead. I wouldn't be surprised if, for instance, bandicoot is waiting for results to come back from the GPU before starting the next operation, whereas the kernels could be enqueued even before they finish

01:03 < rcurtin> but, I haven't seen it in a profiler---at this point I think only you have :)

01:04 < zoq> The numbers aren't too far off, but would be great to close the gap.

01:05 < zoq> if not we can just always compare against TF and PyTorch :)

01:05 < rcurtin> :)

01:05 < rcurtin> there are still many kernels in bandicoot that are totally unoptimized and could be implemented way better

01:05 < rcurtin> so I think it shouldn't be *too* hard to get closer

02:03 < abernauer[m]> rcurtin: I will spend some time this weekend making those changes you suggested to the PR. Just a little drained from 9-4 zoom interviews this week :')

02:04 < abernauer[m]> My knowledge of R and taking James stats 385 class payed off today though.

02:43 < rcurtin> whew, 9-4! that's a lot

02:47 < abernauer[m]> There were breaks today. I had to take an intake form from excel with macros to another technology. Which I had to in a two and a half hour time block while in a call.

11:42 < heisenbug> msg heisenbug identify gopi@1234

12:13 < heisenbug> Hey, anybody active? I wanted to talk about data preprocessing in mlpack...

13:44 < heisenbug> Hello...

13:47 < zoq> heisenbug: I'm currently busy.

13:47 < heisenbug> Oh, its fine...just ping if you get free...

18:02 < AakashkaushikGit> Hey this might just be a silly question but say we have a huge vector that's not going to change it's size while in a for loop. Why don't we store the vector size in a variable, it might just be a very small optimization but say we have 20 loops in a file and a very huge vector of layers as models might get big, maybe 150 layers is a okaish number. will this make any difference ?

18:05 < AakashkaushikGit> I haven't actually read the machine code that the compiler produces, so i am not sure if it caches that vector.size() in this specific case but this just came to my mind

22:06 < zoq> AakashkaushikGit: vector.size() is already a constant, it's not that a call to .size() will calculate the size of the vector; so I don't think this will have an effect, but maybe that wasn't your question?

22:07 < zoq> heisenbug: Maybe it's easier if you ask here and once someone has a change can provide an answer :)

22:54 < AakashkaushikGit> Hi @

22:56 < AakashkaushikGit> Hi @zoq, maybe i wasn't able to frame it right so when we call size() on a vector that is basically something like the end pointer minus the start and returning that right ?, And when we loop thay won't that take more time than assigning that to a temporary variable and using that variable in the loop.

22:56 < AakashkaushikGit> That*

22:57 < AakashkaushikGit> For the loop*

23:17 < abernauer[m]> I got the job!!!

23:28 < zoq> AakashkaushikGit: Haven't looked into the implementation, but if that's the case it will probbably optimized out.

23:28 < zoq> abernauer[m]: Congratulations!