06:14 <JohnPetrucci[m]> Hello everyone! Hope you are all doing well. I had some basic questions regarding testing that I hoping to get to gauge the community's thoughts on. I noticed that for most of the neural network functions, we compare if our results match with the corresponding pytorch function. Is there a better way of testing that doesn't rely on Pytorch because (1) we might have some functions that are not present in PyTorch and (2) Pytorch itself

06:14 <JohnPetrucci[m]> might have the wrong implementations? I feel that relying on another framework for verifying our implementation is probably not the best approach. How do other frameworks like Pytorch, tensor flow, numpy, and Jax test their implementations? Do they hand-compute the results on a particular input and verify that their implementation generates the same results? This approach is prone to human error and gets especially difficult when the

06:14 <JohnPetrucci[m]> functions are too complex. Are there other approaches to testing? Happy to know your

06:14 <JohnPetrucci[m]> * Hello everyone! Hope you are all doing well. I had some basic questions regarding testing that I hoping to get to gauge the community's thoughts on. I noticed that for most of the neural network functions, we compare if our results match with the corresponding pytorch function. Is there a better way of testing that doesn't rely on Pytorch because (1) we might have some functions that are not present in PyTorch and (2) Pytorch itself

06:14 <JohnPetrucci[m]> functions are too complex. Are there other approaches to testing? Would love to know the communitys thoughts on this 😊

06:14 <JohnPetrucci[m]> might have the wrong implementations? I feel that relying on another framework for verifying our implementation is probably not the best approach. How do other frameworks like Pytorch, tensor flow, numpy, and Jax test their implementations? Do they hand-compute the results on a particular input and verify that their implementation generates the same results? This approach is prone to human error and gets especially difficult when the

06:15 <JohnPetrucci[m]> * Hello everyone! Hope you are all doing well. I had some basic questions regarding testing that I was hoping to gauge the community's thoughts on. I noticed that for most of the neural network functions, we compare if our results match with the corresponding pytorch function. Is there a better way of testing that doesn't rely on Pytorch because (1) we might have some functions that are not present in PyTorch and (2) Pytorch itself

06:15 <JohnPetrucci[m]> functions are too complex. Are there other approaches to testing? Would love to know the cocommunity'shoughts on this 😊

06:15 <JohnPetrucci[m]> might have the wrong implementations? I feel that relying on another framework for verifying our implementation is probably not the best approach. How do other frameworks like Pytorch, tensor flow, numpy, and Jax test their implementations? Do they hand-compute the results on a particular input and verify that their implementation generates the same results? This approach is prone to human error and gets especially difficult when the

07:38 <rcurtin[m]> John Petrucci: hi there! In general I agree with your assessment and I personally prefer tests against either known ground-truth (e.g. for a successfully trained model its gradient should be zero, this type of thing). That said, it can sometimes also be useful just to check that the results agree with something from another library. I would say that comparing against TF or PyTorch shouldn't be the *only* test, though. I'm not sure how TF or

07:38 <rcurtin[m]> PyTorch approach the issue of testing, so I can't say much about that. It is worth pointing out that for mlpack there are some "implicit" tests that are not found in `src/mlpack/tests/`; for example, the `examples` repository is also tested, and so if something subtle breaks that doesn't get noticed by the unit tests for mlpack, it may show up as a convergence failure there.

07:40 <rcurtin[m]> Anyway... overall, testing machine learning algorithms is very difficult! My take is that you can often test whether or not things are terribly wrong, but it can sometimes be difficult or impossible to test that everything is perfectly right