12:00 < zoq> Ä

12:01 < zoq> ups :)

13:04 < jenkins-mlpack2> Project docker mlpack nightly build build #124: STILL UNSTABLE in 7 hr 50 min: http://ci.mlpack.org/job/docker%20mlpack%20nightly%20build/124/

14:59 < akhandait> rcurtin: zoq: That's a really good idea!

15:05 < akhandait> I was also going through the issue where a FunctionWrapper is being planned for the optimizer framework (#1481). I was thinking about something like that in summers. I guess Marcus is working on it?

15:50 < zoq> akhandait: I did work on an initial idea: https://github.com/mlpack/ensmallen/pull/49

15:50 < zoq> Happy to collaborate on the idea.

17:13 < akhandait> zoq: I will be happy to help. I just went through the PR. I was also going through the discussion on #1481. I will need some time to fully understand the specifics of this issue. For now, I have understood that we will need to modify all the optimizers as you did in `sarah.hpp`.

17:19 < akhandait> zoq: Another thing I was wondering about, currently we can't implement residual blocks, can we?

17:51 < zoq> akhandait: Right, currently we don't support residual blocks.

18:23 < akhandait> zoq: I think the easiest way to do it with what we currently have is to add a boolean to the Sequential object. If true, all we have to do is to return x + f(x) instead of just f(x).

18:24 < zoq> akhandait: Yeah, I think that's a good idea.

18:26 < akhandait> zoq: Okay, cool then. My exams will end on 22nd. Soon after, I hope to open a PR. The support for residual blocks will be great to have!

18:29 < davida> akhandait: How would you allow the user to link multiple resnet blocks together? You would need to build a block as a submodel, then combine the submodels together with input from the previous submodel (i.e. f(x)) plus the input to the previos submodel.

18:31 < davida> akhandait: and to take this submodel building a step further, could we then use it to build an Inception Netowrk?

18:35 < akhandait> davida: Sorry, I didn't quite get you. What exactly do you mean by a submodel here? If we modify the current Sequential layer object(which can hold the layers of a residual block) to output x + f(x) if the user wants it, we can then stack multiple such Sequential layers to create a resnet.

18:37 < akhandait> davida: About the inception modules, I am not sure yet, but I can think the Concat layer can be used for that.

18:39 < akhandait> zoq: Do you think we can use the Concat layer to make an Inception module?

18:42 < davida> akhandait: It might be possible but I am not yet aware of how to stack the layers into one big model. One ResNet module may consist of multiple layers. (e.g. a simple ResNET block = 3x3x64 Conv. -> 3x3x64 Conv.) which needs to be linked to the next ResNET block = 3x3x128 Conv. -> 3x3x128 Conv.

18:44 < davida> akhandait: To see the level of complexity that is possible in image processing, this link shows the Inception Network. https://hacktilldawn.com/2016/09/25/inception-modules-explained-and-implemented/

18:46 < davida> akhandait: If it was possible to build one large network from smaller subnetworks, defining what should be input to the following layer from the previous layerS (note plural) then this type of "Inception Network" could be possible in MLPACK.

18:48 < akhandait> davida: We will use the Sequential object exactly for that. What it currently does is take multiple layers(for example, 3x3x64 conv -> 3x3x64 conv) and forward pass through them collectively. If we modify it to output the input added to the output(x + f(x)), I think we can build residual blocks

18:49 < davida> akhandait: "Inception Network" is basically a complex ResNET which does not only add f(x)+x but it adds f(x)+g(x)+h(x)... as inputs to the next layer, not just the plain input x from the previous layer.

18:50 < davida> If you take a look at the link I posted you will see a picture of it.

18:50 < akhandait> Now, about the Inception network, I think we can use both the Concat and Sequential objects to implement that.

18:50 < akhandait> Yeah, nice article :)

18:51 < akhandait> Have you tried out those two layers of the ANN module?

18:52 < davida> Concat and Sequential? No. I am only just starting out with MLPACK. Are there some examples somewhere on how to use those correctly to build more complex models?

18:56 < akhandait> davida: Oh sorry, then I might not have made much sense to you. We have tutorials for the ANN module (https://www.mlpack.org/docs/mlpack-git/doxygen/anntutorial.html), but I don't think we have any for those layers yet. Maybe you could just go through this to get familiar with the ANN module of mlpack and then read the documentation of the specific layers. :)

18:58 < davida> I have built FFNs, RNNs and LSTMs so far, but have got stuck at ResNETs and more complex image processing networks. I am trying to convert the Coursera Deeplearnng.ai projects from Python/Numpy/Tensorflow/Keras into C++/Armadillo/MLPACK.

19:00 < davida> I have been thru' that tutorial already but ended up having to go into the tests to figure out how to do more complex stuff. So far I have not come across Sequential layers or Concat layers. Any tests for those I can look at?

19:03 < davida> BTW - I got severely stuck on MLPACK in the beginning as nothing would work on Windows. Everything compiled but none of the models I built converged. Ultimately, once zoq: took some of my code an ran it on his machine successfully, I tried building MLPACK on a Windows 10 Ubuntu shell and everything is working well. I have no idea what is failing on Windows, but I can say that I used Armadillo alone quite well on my Windows code with no problems so it

19:03 < davida> is unlikely to be that library causing problems.

19:04 < rcurtin> davida: I'd really like to fix the windows issues, but I don't have any setup to duplicate Windows problems on

19:04 < rcurtin> if you (or anyone) are willing to provide patches and fixes I think we would all be happy to see them merged

19:05 < davida> I would, if I had any clue on where to start looking for the problem.

19:05 < akhandait> davida: Oh that's great! I think it will be better if you read the documentation of the Sequential and Concat layers. it will help in building more complex networks. Regarding the tests, you can check the tests for Concat layer in ann_layer_test.cpp. I just checked and we don't have any tests for Sequential layer, which is kind of weird.

19:06 < davida> Maybe when you release 3.04 I will re-install on my Windows platform and try my code again to see if that fixes it.

19:20 < davida> OK. I just see that 3.04 is out now. Will try it over the weekend on my platform. Maybe the problems will be fixed as I did not have the latest patches installed previously.

19:20 < rcurtin> davida: sounds good. If you do have problems in the future, please do report bugs---although like I said I don't really have the setup to debug them, maybe someone else in the organization can :)

19:21 < davida> Will do. Now I have two setups running (Ubuntu & Windows) I can compare the code on the two.

19:22 < davida> I will need to upgrade MLPACK on my Ubuntu as well this weekend. The last one I pulled from the git about a week or so ago

20:14 < davida> akhandait: Here is the paper on GoogLeNET, very similar to the Inception Network. Note that there is also the possibility in this network to get output from earlier layers. They have SoftMax0, SoftMax1 & SoftMax2 for outputs. https://arxiv.org/pdf/1409.4842.pdf If we could build that model in MLPACK then that would be the ultimate achievement. (Training might need a larger computer than mine though. =D )

20:20 < zoq> akhandait davida: Nilay started to implement the inception model as an layer and used the concat layer (https://github.com/mlpack/mlpack/pull/757).