<jonpsy[m]>
hmm, looks like we don't have multiplying matrix in batch like ```torch.bmm```
<jonpsy[m]>
or do we?
<heisenbuugGopiMT>
@shrit:matrix.org Small issue. Actually there are some tests failing for `sp_mat`. They are related to formats other than `.csv` I beleive.
<heisenbuugGopiMT>
As you can see the `warning: SpMat::load(): couldn't load from the given stream`
<heisenbuugGopiMT>
Actually I forgot to uncomment all the tests in `load_save_test` so that's why missed these. sorry...
<say4n[m]>
<jonpsy[m]> "hmm, looks like we don't have mu" <- Armadillo [doesn't support](http://arma.sourceforge.net/docs.html#operators) multiplication of two arma::Cube, so you'll need to implement the multiplication for each item in the batch. Iterate with batch as first dim of arma::Cube (b x m x n @ b x n x p = b x m x p)?
<say4n[m]>
(looked at the documentation, I hope it is up to date :))
<jonpsy[m]>
yeap, i was thinking we could omp parallelize it
<jonpsy[m]>
Also, in our replay folder. Should ```states``` be of the type ```std::vector<StateType>``` rather than ```arma::mat```, because if we have stack of image as input then we won't be able to store.
<heisenbuugGopiMT>
But when I am trying to use the build, i.e. run some program, I am getting this exception
<heisenbuugGopiMT>
Any idea why?
<heisenbuugGopiMT>
I made some changes to the. The build was successful and even all the test cases passed.
<heisenbuugGopiMT>
`free(): invalid next size (normal)`
<heisenbuugGopiMT>
* I made some changes to the code, the build was successful and even all the test cases passed.
<heisenbuugGopiMT>
But when I am trying to use the build, i.e. run some program, I am getting this exception
<heisenbuugGopiMT>
`free(): invalid next size (normal)`
<heisenbuugGopiMT>
Any idea why?
<heisenbuugGopiMT>
* I made some changes to the code, the build was successful and even all test cases passed.
<heisenbuugGopiMT>
But when I am trying to use the build, i.e. run some program, I am getting this exception
<heisenbuugGopiMT>
`free(): invalid next size (normal)`
<heisenbuugGopiMT>
Any idea why?
<shrit[m]>
What is the program you are trying to run?
<shrit[m]>
You can try to run the CLI programs too in order to be sure what is happening
<heisenbuugGopiMT>
And this looks different for with and without `datasetmapper`
<shrit[m]>
I mean there is no need for line 5
<shrit[m]>
* I think there is no need for line 5
<shrit[m]>
no, sorry
<shrit[m]>
I mis understood the code
<shrit[m]>
Would you clarify a little bit you question>
<shrit[m]>
?
<heisenbuugGopiMT>
I was planning to make these parts into function, now my doubt is we already have something called NumericParse and CategoricalParse
<heisenbuugGopiMT>
And in both these parse we again have 2 cases where we with either doing it to map where we consider token or just to get size where we don't need to worry about token.
<heisenbuugGopiMT>
Just small difference in both cases.
<heisenbuugGopiMT>
Now should I make different functions for these or like I can add a bool `isNumeric` and using an if condition I can optionally execute the token part.
<shrit[m]>
So, you mean that the difference is only the two loops?
<shrit[m]>
You can put it as a template parameters
<heisenbuugGopiMT>
Ahh, I think I didn't make it clear. Give me a sec.
<shrit[m]>
template<bool Numeric> 👍️
<heisenbuugGopiMT>
Ohh
<heisenbuugGopiMT>
Okay...
<zoq[m]>
<jonpsy[m]> "Also, in our replay folder. Shou" <- When do you have a stack of images, usually you get the observations and perform an action, get the observations perform an action.
<jonpsy[m]>
the added support for multi-objective framework wasn't mentioned :((
<rcurtin[m]>
yeah, unfortunately the review process was super, super long
<rcurtin[m]>
our original submission was May 2020...
<jonpsy[m]>
ah!
<rcurtin[m]>
so I think the hope with a paper like that is just that it piques people's interest (it is only a 4 page summary anyway), and then they go check it out and find out about all the cool stuff it can do 😃
<jonpsy[m]>
we could've compared with ```pymoo``` and ```pagmo``` and other ```moo``` frameworks :)
<rcurtin[m]>
yeah, definitely---maybe we can do this in a follow-up paper or something like this? the 4-page limit was absolutely brutal (especially with the JMLR style) so basically all you can do in 4 pages is say "we made a library, here is a short API snippet, here is a super simple experiment, the license is <X>, go check it out"
<jonpsy[m]>
For sure!
<jonpsy[m]>
infact, Marcus did suggest it, but i was too lazy ;). Benchmarking MOO algorithms is definetely on my TODO list (if i get the time)
<rcurtin[m]>
👍️
<zoq[m]>
> <@jonpsy:matrix.org> > When do you have a stack of images, usually you get the observations and perform an action, get the observations perform an action.
<zoq[m]>
Do we? I think it's just one image, I mean we are in a specific state right?
<zoq[m]>
> so like, in mario we take a stack of images as input right? Which we pass to convolutional network
<zoq[m]>
>
<shrit[m]>
one year for four pages
<shrit[m]>
This is a lot of time
<zoq[m]>
In this case we got some actual feedback, that had to be addressed.
<jonpsy[m]>
> Do we? I think it's just one image, I mean we are in a specific state right?
<jonpsy[m]>
yes, but remember a single image doesn't tell much. So we stack few consecutive images into 1 and then call it s_t
<zoq[m]>
But how do you get the observation from the future without an action?
<zoq[m]>
you can pass multiple cols to the conv layer.
<jonpsy[m]>
> I guess you're suggesting you're storing each image as vector?
<jonpsy[m]>
so, this?
<zoq[m]>
Yes, at least this is what we are currently doing.
<jonpsy[m]>
hm, but to pass to ```CNN``` each image need to be a vector right?
<jonpsy[m]>
* hm, but to pass to `CNN` each image need to be a matrix* right?
<zoq[m]>
KInda, we pass a matrix to the CNN, where each col represents one image, internally the conv layer unpacks the matrix into a `cube` to make the code easier to understand, which means transforming the col into a matrix.
<jonpsy[m]>
that kinda sucks ngl
<jonpsy[m]>
why go this far?
<jonpsy[m]>
i guess its because we cant support 4D tensor?
<zoq[m]>
To have a unified interface, I mean what is the downside?
<zoq[m]>
This way we can use `arma::mat` for every method, can easily transfer data between different methods.
<zoq[m]>
We could make an interface for `arma::cube` armadillo supports that.
<zoq[m]>
But what is the point in doing that.
<rcurtin[m]>
in the `ann-vtable` branch, some things are being changed around. our fundamental issue is that Armadillo doesn't have support for higher-order tensors
<jonpsy[m]>
exactly!
<rcurtin[m]>
so, I think the API we have to settle on is that a feedforward network takes an `arma::mat`; in this `arma::mat`, each column corresponds to a data point, so the dimensions are `(n_features, batch_size)`
<rcurtin[m]>
then, we have a function `InputDimensions()` that allows you to pass a `std::vector<size_t>` that specifies the actual dimensions to interpret each column as
<rcurtin[m]>
so, for instance, if you have an image, you might set `InputDimensions()` to `{ width, height, channels }`
<rcurtin[m]>
and then, the `arma::mat` you pass to `Evaluate()` or `Train()` will have dimensions `(width * height * channels, batchSize)`
<zoq[m]>
Sure, but we still pass a matrix that contains all the images.
<zoq[m]>
Which is what jonpsy said "that kinda sucks ngl"
<zoq[m]>
I'm not sure I agree on that because I don't see the problem.
<rcurtin[m]>
yeah, my thought is that it's not perfect, but with `InputDimensions()` it cleans it up a decent bit
<jonpsy[m]>
rcurtin[m]: question: We handle the image unrolling and re-construction right? Because there's a good chance that it converting to ```vec``` and then retreiving back to ```mat``` might mess up the image due to reshape
<rcurtin[m]>
at least I don't see any realistic alternative. we can write some wrappers so if you have, e.g., a `std::vector<>` of some images or something, we can pack them into an `arma::mat` and get a `std::vector<size_t>` of input dimensions
<zoq[m]>
I mean what is the difference between using `vector<mat> data; data[i]` instead of `mat data; data.col(i)`
<rcurtin[m]>
jonpsy: yeah, right now the user has to handle that, but I think we can provide utilities that do this conversion correctly and handle most cases
<jonpsy[m]>
yep, that sucks
<zoq[m]>
We already do, if you use the `data::Load` function it will handle that for you.
<rcurtin[m]>
I mean, you might be better off here if you consider the constraints and consider what we can do to make things work within those constraints
<rcurtin[m]>
I think zoq is right, we have some loading functionality for images. I'm not sure exactly the format it packs things into, but it wouldn't be too hard to modify it to return a packed `arma::mat` plus a `std::vector<size_t>` of dimensions
<jonpsy[m]>
yeah, given the constraints. This is nice
<rcurtin[m]>
well, that's still not a very positive outlook either ;)
<rcurtin[m]>
given those constraints, I believe we can make things a little nicer, we just have to consider those constraints in figuring out what to do
<rcurtin[m]>
imagine a C++ dataframe class that actually does have support for higher dimensions, and just wraps an `arma::mat`
<rcurtin[m]>
it doesn't support higher-order tensor operations, but maybe it provides an interface so you can do, e.g., `image.at(x, y, z, channel)` and get a value
<heisenbuugGopiMT>
I was somewhat thinking that this discussion will lead to dataframe
<rcurtin[m]>
then, you can imagine the `FFN` taking in one of these dataframe-like classes, setting `InputDimensions()` automatically, and learning on the underlying `arma::mat` representation
<jonpsy[m]>
sounds like a plan!
<zoq[m]>
Hm, in most of the cases you don't actually need to reshape, it's more the other way around, in case of the conv layer we reshape internally, because it's easier to read but in the end it's a vector operation.
<shrit[m]>
The issue is related to manipulating the `arma::cube` right?
<rcurtin[m]>
the other comment worth pointing out here is that it's not always the best choice to imitate the design choices of more popular toolkits. there is an advantage to that, which is that it is a familiar interface to users, but, in many cases, popular toolkits have really counterintuitive and complicated APIs (one decent example is TensorFlow's original API)
<zoq[m]>
rcurtin[m]: In this case it always comes down to the actually implementation, in some cases it makes sense to interpret the data as a matrix in some cases it makes sense to use a vector.
<zoq[m]>
Like passing a matrix to the conv layer makes sense, but passing it to e.g. a relu activation function a vector would be better.
<rcurtin[m]>
ah, change "in many cases" to "in some cases". I don't want to overstate it 😃 but, it is worth keeping in mind when considering API design that sometimes it is a good thing to explicitly choose something different than the 'typical' approach
<rcurtin[m]>
in our convolution layer, what it really does at the moment is take in an `arma::vec` representing a single image/input, then "reinterprets" it as something of the correct dimensions. other layers that also expect higher-dimensional inputs do the same thing
<rcurtin[m]>
that, too, is perhaps a thing we could improve upon 😃
<rcurtin[m]>
the only issue is the amount of effort it takes to do so 😃
<jonpsy[m]>
yep! That was exactly my concern
<jonpsy[m]>
the re-shape to matrix thing
<jonpsy[m]>
and potential mess up when re-constructing
<rcurtin[m]>
internally, that's what utility functions and unit tests are good for :) but for a user, I agree, we can do better than we're currently doing
<zoq[m]>
It's just a reinterpretation of the input.
<jonpsy[m]>
hm, so its using the same memory space
<jonpsy[m]>
like ```torch.view```
<rcurtin[m]>
yeah, exactly, pretty much any tensor library or matrix library will work exactly like this. it allocates a big block of memory, and then simply partitions it logically for each dimension
<zoq[m]>
Right, no copy involved, it's really just a reinterpretation because people like to thing about images as a matrix.
<zoq[m]>
* Right, no copy involved, it's really just a reinterpretation because people like to think about images as a matrix.
<jonpsy[m]>
i suppose that operation guarantees that the image would be re-constructed as was originally intended
<rcurtin[m]>
assuming the user has the dimensions specified in the same "order" as we reconstruct them in, yeah
<rcurtin[m]>
(and, technically, if the user passes in something as `{height, width, channels}`, so long as they specify their convolution filter shape as `{height, width}`, too, then everything will work out fine)
<jonpsy[m]>
okay so the ```reshape`` and "potential mess up" thing is cleared
<jonpsy[m]>
* okay so the ```reshape``` and "potential mess up" thing is cleared
<jonpsy[m]>
now we're counting on user to correctly convert image to ```vec```
<zoq[m]>
Right, if you stay in the mlpack universe and use the `data::Load` function that is already handled for you.
<zoq[m]>
But yeah, we don't provide a utility function that does that for you.
<rcurtin[m]>
and maybe someday this magical dataframe class will exist and provide a nice interface on top of it 😃 (maybe XTensor could do the job, but we'd still need to integrate it)
<jonpsy[m]>
zoq[m]: sounds like a PR :)
<jonpsy[m]>
xtensor's pretty cool
<jonpsy[m]>
well, this was very educational!
<rcurtin[m]>
:) another nice thing about xtensor and most matrix/tensor libraries using a big block of memory, is that it would theoretically be possible to take an xtensor dataframe and wrap the underlying block of memory in an `arma::mat` without a copy
<rcurtin[m]>
this is actually what is done to transition between Armadillo and Numpy matrices for the Python bindings
<rcurtin[m]>
(and same thing in Go, R, Julia, and I think the Java bindings PR too but I haven't had any time to return to that :( ...)
<jonpsy[m]>
rcurtin[m]: hm, so you'll take the ```mem_ptr``` and use advanced_ctor
<rcurtin[m]>
yeah, exactly
<jonpsy[m]>
we could wrap it in a function saying ```xTensorToMat```
<jonpsy[m]>
ez
<rcurtin[m]>
right, exactly, I think it would be straightforward (but perhaps not trivial) to create a tool like that
<rcurtin[m]>
I'm still waiting on an email from Sylvain Corlay (CEO of QuantStack, who produces xtensor) to see what his level of interest in a collaboration is
<rcurtin[m]>
(apparently it must not be high enough to respond quickly 😃 but I'm sure we'll hear from him eventually. I know he is interested in C++ data science, so if someone who works on XTensor wants to collaborate on mlpack integration, I think it could be a pretty nice project)
<jonpsy[m]>
there's also ArrayFIre
<rcurtin[m]>
they're actually based in Atlanta---I did a job interview with them at some point but didn't end up going that direction
<rcurtin[m]>
it seems like they have done a lot of work recently and gotten some press, but, I haven't checked in. at least in 2015, it didn't seem like there was much momentum
<shrit[m]>
rcurtin: it is vacation time, everyone on vacation in France now
<rcurtin[m]>
ahhhh, I see 😃
<jonpsy[m]>
they're super popular since Pytorch is using ArrayFire in their C++ API
<shrit[m]>
vacations are from 14 July to 15 August, and sometime to 23 of Augugst
<rcurtin[m]>
the complexity for ArrayFire is that they are built on GPUs---and Armadillo can't just wrap GPU memory
<rcurtin[m]>
but, presumably one could wrap a Bandicoot matrix around an ArrayFire `array` (but there is still a lot of work to be done to make bandicoot work in general)
<shrit[m]>
* vacations are from 14 July to 15 August, and sometime to 23 of August
<jonpsy[m]>
i see.
<jonpsy[m]>
this was fun. Thanks for all the details reg. conv neural network, i couldn't have figued in a 100 years
<rcurtin[m]>
100 years is a long time, I am sure you could have figured it out by then 😃
<rcurtin[m]>
I enjoyed the discussion too---it's always good to talk about the larger picture and how we can improve things 👍️
<heisenbuugGopiMT>
@shrit:matrix.org, I pushed the code.
<heisenbuugGopiMT>
I will add some comments, working on it.
<heisenbuugGopiMT>
But I think you can have a look now.
<heisenbuugGopiMT>
Only failing tests are related to `sp_mat`
<shrit[m]>
Is there are any new failing tests?
<heisenbuugGopiMT>
Nope, nothing new, just those stream related issues...
<heisenbuugGopiMT>
That also for formats other than `csv`
<shrit[m]>
The good looks good, much better than before, thanks for the organization
<shrit[m]>
If you would like I can review it now, or in a couple of hour when you finish all the modifications
<shrit[m]>
Let me know what do you think :+1:
<heisenbuugGopiMT>
I am my dinner now, although it's late...
<heisenbuugGopiMT>
I will take an hour to get back to work, so up to you...
<shrit[m]>
* Let me know what do you think 👍️
<shrit[m]>
Okay, I will review it at the end of the day. You can incorporate my modifications tomorrow